multitasking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "91a5a6c6-c46d-4d7b-bfc4-41ab229a54a3",
   "metadata": {},
   "source": [
    "# Multitasking Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98de986-b174-4c88-8b6e-9cd606a9c816",
   "metadata": {},
   "source": [
    "I/O bound processes are a typical example of advantages of concurrent/parallel processing over sequential execution.\n",
    "Networking is a very common I/O bound task and is well suited for explaining different approaches to this problem.\n",
    "Let's use link checker application as a real world example of I/O bound task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f1e38f-c088-4033-8a72-d2ca5f769f62",
   "metadata": {},
   "source": [
    "## Link Checker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0674d2e2-af96-4800-884b-bd0337598b7b",
   "metadata": {},
   "source": [
    "First, we need to have a list of links to check. It doesn't really matter what they are, so let me extract them from Wikipedia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ebed15c6-f880-4916-8c10-a77018658aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcfe0d77-a63f-4510-a9b8-ebc88c0b6c8d",
   "metadata": {},
   "source": [
    "I've been recently checking a [List of Nobel prize winners in Chemistry](https://en.wikipedia.org/wiki/List_of_Nobel_laureates_in_Chemistry), and there are a lot of links we can use for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dcdc0e9d-aa1e-4780-84f0-864010b0b849",
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki_url = 'https://en.wikipedia.org/wiki/List_of_Nobel_laureates_in_Chemistry'\n",
    "r = requests.get(wiki_url)\n",
    "soup = BeautifulSoup(r.text, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819bfd07-4af4-4dac-b18c-ef48b19826aa",
   "metadata": {},
   "source": [
    "The main table on this page has a link for every Nobel laureate, country, etc. Let's use these links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3e9680e7-851c-45b9-8301-ff99eae9c78f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "table = soup.table\n",
    "links = []\n",
    "for link in table.find_all('a'):\n",
    "    links.append(link.get('href'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec6dd819-afbf-4539-bffb-70b717c0d612",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['#endnote_1',\n",
       " '#endnote_2',\n",
       " '#endnote_3',\n",
       " '/wiki/File:Vant_Hoff.jpg',\n",
       " '/wiki/Jacobus_Henricus_van_%27t_Hoff',\n",
       " '/wiki/Netherlands',\n",
       " '/wiki/Chemical_dynamics',\n",
       " '/wiki/Osmotic_pressure',\n",
       " '#cite_note-19',\n",
       " '/wiki/File:Hermann_Emil_Fischer_c1895.jpg']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7101118f-d874-49c3-9bed-f54529f02e97",
   "metadata": {},
   "source": [
    "There is a mix of useful links I want to check like '/wiki/Chemical_dynamics' and references to pictures etc. which I don't want to use. We can extract useful links using regular expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af39b79c-c443-4a4e-b0d4-dcb72af7b5c8",
   "metadata": {},
   "source": [
    "### How to regex?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d33da628-5f94-420d-ad75-eba18a65067e",
   "metadata": {},
   "source": [
    "For quick prototypes like this one, my usual process of writing regexes is the following:  \n",
    "1. Get examples of text from which you want to extract information. Here's it's a `link[:10]` from above.\n",
    "2. Go to [regex101](https://regex101.com/) and copy your examples.\n",
    "3. Write your regex using Quick Reference from regex101. The site shows matches interactively which makes writing regex fast.\n",
    "4. Use created pattern in Python with [re](https://docs.python.org/3/library/re.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6baae64-f1df-4856-85dc-783544be7e17",
   "metadata": {},
   "source": [
    "![regex](images/regex.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "29fe1787-786d-415f-bc92-710982f5e4c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "pattern = re.compile('\\/wiki\\/\\w+$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3021f6ed-99ee-4621-b44d-d6abe5171c31",
   "metadata": {},
   "outputs": [],
   "source": [
    "good_links = [link for link in links if pattern.match(link)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f4fb7e6a-11a7-4aab-b5f9-0d364e836b58",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = 'https://en.wikipedia.org/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a9985f1f-e682-402d-ba48-c1f97d7ceab6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "full_links = [base_url + link for link in good_links]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "03b2bea6-bf67-4f40-89d0-d00b2a91bdad",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://en.wikipedia.org//wiki/Netherlands',\n",
       " 'https://en.wikipedia.org//wiki/Chemical_dynamics',\n",
       " 'https://en.wikipedia.org//wiki/Osmotic_pressure',\n",
       " 'https://en.wikipedia.org//wiki/Hermann_Emil_Fischer',\n",
       " 'https://en.wikipedia.org//wiki/German_Empire',\n",
       " 'https://en.wikipedia.org//wiki/Svante_Arrhenius',\n",
       " 'https://en.wikipedia.org//wiki/Sweden',\n",
       " 'https://en.wikipedia.org//wiki/Electrolyte',\n",
       " 'https://en.wikipedia.org//wiki/William_Ramsay',\n",
       " 'https://en.wikipedia.org//wiki/United_Kingdom']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_links[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f82adb4-60d6-4235-b603-7ca820e56c7d",
   "metadata": {},
   "source": [
    "Let's see how long it roughly takes to check all of these links. I want it to be long enough to see differnces between methods, but no need to wait forever. Around 1 minute should be enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b2a0c4cf-6554-43e0-96a1-4c0f795ed83c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ba2a612d-794d-45db-b225-f5cce1226655",
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "for link in full_links:\n",
    "    r = requests.get(link)\n",
    "end = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9e08cea0-9de2-4d6d-87e4-d10c6dd8c22f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 117.46 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f'Took {end - start:.2f} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36dc71d-ffb2-4ad0-b572-c43c9eb655ac",
   "metadata": {},
   "source": [
    "Ok, that's a bit too long, so let's get rid of 3/4 of the links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0faa5e1b-c335-4ddf-bd60-b1b2b7a43d40",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "full_links = random.sample(full_links, round(0.25*len(full_links)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0bef30e0-3773-4848-b814-377b9060cf4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "for link in full_links:\n",
    "    r = requests.get(link)\n",
    "end = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dc3c190b-035c-420a-8559-b9676c39a5f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 29.21 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f'Took {end - start:.2f} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02058a38-da56-4525-9041-6b92e4b5cd6d",
   "metadata": {},
   "source": [
    "Long enough to see the difference between various methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "574ab232-1020-464b-95e3-32115f71a1aa",
   "metadata": {},
   "source": [
    "Let's break 10% of the links to simulate the task we want to perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7a216b59-a339-4bc5-9bbc-bae46e537eb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prep_test_links(links):\n",
    "    to_break = random.sample(links, round(0.1*len(links)))\n",
    "    broken = [link.replace('kipedia', 'asdf') for link in to_break]\n",
    "    broken_count = len(broken)\n",
    "    ok_count = len(links)\n",
    "    test_links = links + broken\n",
    "    random.shuffle(test_links)\n",
    "    return test_links, broken_count, ok_count"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f7a9f6-c130-4181-8bea-d317d5fc91ab",
   "metadata": {},
   "source": [
    "## Sequential check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944620d6-95e4-40c3-b5ef-fca5f6593328",
   "metadata": {},
   "source": [
    "We can access links one-by-one and check how long it takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d1a98f37-bd37-4d95-8428-4e1d639dd296",
   "metadata": {},
   "outputs": [],
   "source": [
    "from requests.exceptions import ConnectionError\n",
    "\n",
    "def is_working_link(link):\n",
    "    try:\n",
    "        r = requests.get(link)\n",
    "    except ConnectionError:\n",
    "        return False\n",
    "    else:\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4b0af76c-b91c-4d9a-b8be-09bd84dcf8e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq_check_links(links):\n",
    "    ok_count = 0\n",
    "    broken_count = 0\n",
    "    for link in links:\n",
    "        if is_working_link(link):\n",
    "            ok_count += 1\n",
    "        else:\n",
    "            broken_count += 1\n",
    "            \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "670a7cc6-5b85-4942-ab5e-2921fcd859ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = seq_check_links(test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5de93900-3ba6-48d2-9436-423cb47a7b16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.57 s, sys: 430 ms, total: 6 s\n",
      "Wall time: 29.5 s\n"
     ]
    }
   ],
   "source": [
    "%time seq_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb95a41c-0703-4535-8ef0-70dfb493af07",
   "metadata": {},
   "source": [
    "We can also see that while running this function takes ~30s, CPU is used only for ~5s. All the rest is spent waiting for I/O, which suggests that we should see a significant speed up from using other methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a21e909-a127-4393-988b-4bdc803b7c5d",
   "metadata": {},
   "source": [
    "## Multithreaded check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d452dc27-7741-4cfc-8e2f-479f5b05739e",
   "metadata": {},
   "source": [
    "For multhithreaded check I just want to make the same requests from different threads. We could use `threading` module for that, but instead I'm going to use `concurrent.futures` module. It provides consistent interface for running both threads and processes which is going to make the comparison clearer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "665a0bf8-a4e2-4878-ad83-bf06428a9949",
   "metadata": {},
   "outputs": [],
   "source": [
    "from concurrent.futures import ThreadPoolExecutor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de0fbeb-92ad-477d-b960-33e58a6d2bcf",
   "metadata": {},
   "source": [
    "The usual way of writing multithreaded code is to create function that processes one item and map it over a sequence of items to process. For sequential check I could have writien `check_links` function without `is_working_link` function, but I'm using it to make it more similar to what's required for multhithreading. We could even write sequential check using `map` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0bd9a250-0134-4313-8f7c-86f6338e1f12",
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq_check_links(links):\n",
    "    ok_count = sum(map(is_working_link, links))\n",
    "    broken_count = len(links) - ok_count\n",
    "            \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1a3f5b3a-c5e2-4793-a7e5-7028bfb3ed5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def seq_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = seq_check_links(test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9c58d3d1-64d4-45a9-b94c-dd43161a8249",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.51 s, sys: 419 ms, total: 5.93 s\n",
      "Wall time: 31.1 s\n"
     ]
    }
   ],
   "source": [
    "%time seq_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1983b350-e9cb-4508-8dc9-fac96410d39c",
   "metadata": {},
   "source": [
    "Again we see that sequential check with `map` takes ~30s, but this version will make comparison clearer so, let's keep it this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "06e71792-f65b-48af-9e70-556fa5c6815b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def thread_check_links(links):\n",
    "    with ThreadPoolExecutor(max_workers=2) as executor:\n",
    "        ok_count = sum(executor.map(is_working_link, links))\n",
    "    broken_count = len(links) - ok_count\n",
    "            \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "beb79eae-91a9-4057-bf6e-403886d6b353",
   "metadata": {},
   "outputs": [],
   "source": [
    "def thread_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = thread_check_links(test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9e5dca31-fb03-42bc-bf7a-ef451cfe163c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.13 s, sys: 414 ms, total: 5.55 s\n",
      "Wall time: 14.8 s\n"
     ]
    }
   ],
   "source": [
    "%time thread_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5526e71-9782-44f5-9030-9601d3634b5c",
   "metadata": {},
   "source": [
    "Using 2 threads made checking links roughly twice as fast!  \n",
    "But why stop there? Or in general how to know how many threads we should use?  \n",
    "As with many other things in programming it's better to not make assumptions, but just test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "dcaa4cd5-29e9-4857-9ec9-c555c08d540b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def thread_check_links(num_threads, links):\n",
    "    with ThreadPoolExecutor(max_workers=num_threads) as executor:\n",
    "        ok_count = sum(executor.map(is_working_link, links))\n",
    "    broken_count = len(links) - ok_count\n",
    "\n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ab58635b-ad93-4484-8b5a-4563481877a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def thread_test_performance():\n",
    "    \n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    \n",
    "    num_threads = []\n",
    "    times = []\n",
    "    for n in (2**n for n in range(1, 7)):\n",
    "        start = time.time()\n",
    "        broken_count, ok_count = thread_check_links(n, test_links)\n",
    "        assert test_broken == broken_count\n",
    "        assert test_ok == ok_count\n",
    "        end = time.time()\n",
    "        \n",
    "        num_threads.append(n)\n",
    "        times.append(end-start)\n",
    "    return num_threads, times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "04608f4b-7e95-4947-bc6c-a93fe9e09e2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_threads, times = thread_test_performance()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5d6fbb7d-f954-4e7f-ab8a-4d844d9a2adf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPXUlEQVR4nO3df2xdZ33H8fdnSRhuYbisXtekZek2ZIT40SCLwUCMtUCyrqIR4g+qMZWtUv5hW5lQqmZIoP3Fpkz8kDYxRVCKtKqbVkJA1UboChOaxMqcpiVpgymDDuK0xBXz2Jg10vDdH76uHDeJf9xr3/vY75d0de95zrHP92lPPz59znPOTVUhSWrPz/S7AEnSyhjgktQoA1ySGmWAS1KjDHBJatTmtdzZ5ZdfXtu3b1/LXUpS844cOfJ0VY0sbF/TAN++fTvj4+NruUtJal6S/zhfu0MoktQoA1ySGmWAS1KjDHBJapQBLkmNWtNZKCtx6Ogk+w9PcGp6hq3DQ+zdOcruHdv6XZYk9d1AB/iho5PsO3iMmTNnAZicnmHfwWMAhrikDW+gh1D2H554NrznzJw5y/7DE32qSJIGx0AH+KnpmWW1S9JGMtABvnV4aFntkrSRDHSA7905ytCWTee0DW3ZxN6do32qSJIGx0BfxJy7UOksFEl6roEOcJgNcQNbkp5roIdQJEkXZoBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSoRQM8yZ1JTic5fp51709SSS5fnfIkSReylDPwu4BdCxuTXA28Dfhej2uSJC3BogFeVV8FfnieVR8Fbgeq10VJkha3ojHwJDcBk1X1yBK23ZNkPMn41NTUSnYnSTqPZQd4kkuAPwE+uJTtq+pAVY1V1djIyMhydydJuoCVnIH/CnAN8EiSJ4CrgIeS/GIvC5MkXdyyHydbVceAX5hb7oT4WFU93cO6JEmLWMo0wnuArwGjSU4muXX1y5IkLWbRM/CqunmR9dt7Vo0kacm8E1OSGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqOW8p2YdyY5neT4vLb9Sb6Z5BtJPpdkeFWrlCQ9x1LOwO8Cdi1oux94RVW9CvgWsK/HdUmSFrFogFfVV4EfLmj7UlU901n8V+CqVahNknQRvRgD/33gHy+0MsmeJONJxqempnqwO0kSdBngST4APAPcfaFtqupAVY1V1djIyEg3u5MkzbN5pT+Y5D3AjcD1VVU9q0iStCQrCvAku4Dbgd+oqv/tbUmSpKVYyjTCe4CvAaNJTia5FfhL4IXA/UkeTvLXq1ynJGmBRc/Aq+rm8zR/ahVqkSQtg3diSlKjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY1aypca35nkdJLj89penOT+JI933i9b3TIlSQst5Qz8LmDXgrY7gAeq6qXAA51lSdIaWjTAq+qrwA8XNN8EfKbz+TPA7t6WJUlazErHwK+oqic7n58CrrjQhkn2JBlPMj41NbXC3UmSFur6ImZVFVAXWX+gqsaqamxkZKTb3UmSOlYa4D9IciVA5/1070qSJC3FSgP8C8Atnc+3AJ/vTTmSpKVayjTCe4CvAaNJTia5Ffgz4K1JHgfe0lmWJK2hzYttUFU3X2DV9T2uRZK0DN6JKUmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUV0FeJI/TvJokuNJ7kny/F4VJkm6uBUHeJJtwB8BY1X1CmAT8K5eFSZJurhuh1A2A0NJNgOXAKe6L0mStBQrDvCqmgT+Avge8CTwX1X1pYXbJdmTZDzJ+NTU1MorlSSdo5shlMuAm4BrgK3ApUnevXC7qjpQVWNVNTYyMrLySiVJ5+hmCOUtwHeraqqqzgAHgV/vTVmSpMV0E+DfA16X5JIkAa4HTvSmLEnSYroZA38QuBd4CDjW+V0HelSXJGkRm7v54ar6EPChHtUiSVoG78SUpEYZ4JLUqK6GUAbJoaOT7D88wanpGbYOD7F35yi7d2zrd1mStGrWRYAfOjrJvoPHmDlzFoDJ6Rn2HTwGYIhLWrfWxRDK/sMTz4b3nJkzZ9l/eKJPFUnS6lsXAX5qemZZ7ZK0HqyLAN86PLSsdklaD9ZFgO/dOcrQlk3ntA1t2cTenaN9qkiSVt+6uIg5d6HSWSiSNpJ1EeAwG+IGtqSNZF0MoUjSRmSAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUqK4CPMlwknuTfDPJiSSv71VhkqSL6/ZOzI8DX6yqdyZ5HnBJD2qSJC3BigM8yYuANwHvAaiqnwA/6U1ZkqTFdDOEcg0wBXw6ydEkn0xy6cKNkuxJMp5kfGpqqovdSZLm6ybANwOvAT5RVTuAHwN3LNyoqg5U1VhVjY2MjHSxO0nSfN0E+EngZFU92Fm+l9lAlyStgRUHeFU9BXw/ydy3JlwPPNaTqiRJi+p2FsofAnd3ZqB8B/i97kuSJC1FVwFeVQ8DY70pRZK0HN6JKUmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDWq6wBPsinJ0ST39aIgSdLS9OIM/DbgRA9+jyRpGboK8CRXAb8NfLI35UiSlqrbM/CPAbcDP73QBkn2JBlPMj41NdXl7iRJczav9AeT3AicrqojSd58oe2q6gBwAGBsbKxWur+1dujoJPsPT3Bqeoatw0Ps3TnK7h3b+l2WJD1rxQEOvAF4e5IbgOcDP5fkb6rq3b0prX8OHZ1k38FjzJw5C8Dk9Az7Dh4DMMQlDYwVD6FU1b6quqqqtgPvAr68HsIbYP/hiWfDe87MmbPsPzzRp4ok6bmcB34ep6ZnltUuSf3QkwCvqn+uqht78bsGwdbhoWW1S1I/eAZ+Hnt3jjK0ZdM5bUNbNrF352ifKpKk5+rmIua6NXeh0lkokgaZAX4Bu3dsM7AlDTSHUCSpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWrUigM8ydVJvpLksSSPJrmtl4VJki6umy90eAZ4f1U9lOSFwJEk91fVYz2qTZJ0ESs+A6+qJ6vqoc7n/wZOAH6FjSStkZ58pVqS7cAO4MHzrNsD7AF4yUte0ovdbXiHjk76fZ2Sur+ImeQFwGeB91XVjxaur6oDVTVWVWMjIyPd7m7DO3R0kn0HjzE5PUMBk9Mz7Dt4jENHJ/tdmqQ11lWAJ9nCbHjfXVUHe1OSLmb/4Qlmzpw9p23mzFn2H57oU0WS+qWbWSgBPgWcqKqP9K4kXcyp6ZlltUtav7o5A38D8LvAdUke7rxu6FFduoCtw0PLape0fnUzC+VfqipV9aqqurbz+odeFqfn2rtzlKEtm85pG9qyib07R/tUkaR+6cksFK2dudkmzkKRZIA3aPeObQa2JJ+FIkmtMsAlqVEGuCQ1yjFwbRg+gkDrjQGuDWHuEQRzd7HOPYIAMMTVLIdQtCH4CAKtRwa4NgQfQaD1yCEUbQhbh4eYPE9Y+wgCrbbVvPbiGbg2BB9BoH5Y7cc/G+DaEHbv2MaH3/FKtg0PEWDb8BAffscrvYCpVbXa114cQtGG4SMItNZW+9qLZ+CStEpW+/HPBrgkrZLVvvbiEIokrZLVfvyzAS5Jq2g1r704hCJJjTLAJalRXQV4kl1JJpJ8O8kdvSpKkrS4FQd4kk3AXwG/BbwcuDnJy3tVmCTp4ro5A38t8O2q+k5V/QT4W+Cm3pQlSVpMN7NQtgHfn7d8Evi1hRsl2QPs6Sz+T5KF95BeDjzdRR2DYD30AdZHP+zDYLAPvfVL52tc9WmEVXUAOHCh9UnGq2pstetYTeuhD7A++mEfBoN9WBvdDKFMAlfPW76q0yZJWgPdBPi/AS9Nck2S5wHvAr7Qm7IkSYtZ8RBKVT2T5A+Aw8Am4M6qenQFv+qCwysNWQ99gPXRD/swGOzDGkhV9bsGSdIKeCemJDXKAJekRvU1wFu8FT/JnUlOJzk+r+3FSe5P8njn/bJ+1riYJFcn+UqSx5I8muS2Tnsz/Ujy/CRfT/JIpw9/2mm/JsmDnWPq7zoX2Adakk1Jjia5r7PcVB+SPJHkWJKHk4x32po5lgCSDCe5N8k3k5xI8voW+tC3AG/4Vvy7gF0L2u4AHqiqlwIPdJYH2TPA+6vq5cDrgPd2/tm31I//A66rqlcD1wK7krwO+HPgo1X1q8B/Arf2r8Qluw04MW+5xT78ZlVdO2/edEvHEsDHgS9W1cuAVzP772Pw+1BVfXkBrwcOz1veB+zrVz3LrH07cHze8gRwZefzlcBEv2tcZn8+D7y11X4AlwAPMXsn8NPA5k77OcfYIL6YvX/iAeA64D4gDfbhCeDyBW3NHEvAi4Dv0pnU0VIf+jmEcr5b8Vv9xtkrqurJzuengCv6WcxyJNkO7AAepLF+dIYeHgZOA/cD/w5MV9UznU1aOKY+BtwO/LSz/PO014cCvpTkSOfRGdDWsXQNMAV8ujOU9ckkl9JAH7yI2WM1++e6ibmZSV4AfBZ4X1X9aP66FvpRVWer6lpmz2JfC7ysvxUtT5IbgdNVdaTftXTpjVX1GmaHQ9+b5E3zVzZwLG0GXgN8oqp2AD9mwXDJoPahnwG+nm7F/0GSKwE676f7XM+ikmxhNrzvrqqDnebm+gFQVdPAV5gdbhhOMneD2qAfU28A3p7kCWaf5nkds2OxLfWBqprsvJ8GPsfsH9OWjqWTwMmqerCzfC+zgT7wfehngK+nW/G/ANzS+XwLs2PKAytJgE8BJ6rqI/NWNdOPJCNJhjufh5gdwz/BbJC/s7PZQPehqvZV1VVVtZ3Z4//LVfU7NNSHJJcmeeHcZ+BtwHEaOpaq6ing+0nmvir+euAxWuhDny8e3AB8i9mxyw/0+4LAEmu+B3gSOMPsX+5bmR23fAB4HPgn4MX9rnORPryR2f8d/AbwcOd1Q0v9AF4FHO304TjwwU77LwNfB74N/D3ws/2udYn9eTNwX2t96NT6SOf16Nx/xy0dS516rwXGO8fTIeCyFvrgrfSS1CgvYkpSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1Kj/B4IKwIm/5lTXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(num_threads, times)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c35ca7e0-2b70-4fac-9071-220e7594aecd",
   "metadata": {},
   "source": [
    "Each new thread takes some memory, so it doesn't make sense to create more of them for marginal speedup. \n",
    "For the final test let's use 16 threads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "2377bfc5-b120-4bd5-81ed-28af9902082c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def thread_check_links(links):\n",
    "    with ThreadPoolExecutor(max_workers=16) as executor:\n",
    "        ok_count = sum(executor.map(is_working_link, links))\n",
    "    broken_count = len(links) - ok_count\n",
    "            \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "895a1866-c8b9-4931-97e8-7408fcdb9fcb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.76 s, sys: 190 ms, total: 1.95 s\n",
      "Wall time: 1.73 s\n"
     ]
    }
   ],
   "source": [
    "%time thread_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98551546-a191-4df7-ab37-748b12abdf91",
   "metadata": {},
   "source": [
    "## Multiprocessing check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73055e8-8d0f-4669-afe5-42a784b31501",
   "metadata": {},
   "source": [
    "`concurrent.futures` library allows use to switch single line of code to move from multithreading to multiprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f19599fb-3fec-426b-9e94-d6d9d3a9958a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from concurrent.futures import ProcessPoolExecutor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "78effc97-b91b-4668-9e87-5bdf0d46c575",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiprocess_check_links(links):\n",
    "    with ProcessPoolExecutor(max_workers=2) as executor:\n",
    "        ok_count = sum(executor.map(is_working_link, links))\n",
    "    broken_count = len(links) - ok_count\n",
    "            \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f600ea6e-63f6-47bb-8e18-b65044249717",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiprocess_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = multiprocess_check_links(test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d4ede1d3-1bc7-42ec-9af2-fe00bf83d96f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process SpawnProcess-1:\n",
      "Traceback (most recent call last):\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/process.py\", line 233, in _process_worker\n",
      "    call_item = call_queue.get(block=True)\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py\", line 116, in get\n",
      "    return _ForkingPickler.loads(res)\n",
      "AttributeError: Can't get attribute 'is_working_link' on <module '__main__' (built-in)>\n",
      "Process SpawnProcess-2:\n",
      "Traceback (most recent call last):\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/process.py\", line 233, in _process_worker\n",
      "    call_item = call_queue.get(block=True)\n",
      "  File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py\", line 116, in get\n",
      "    return _ForkingPickler.loads(res)\n",
      "AttributeError: Can't get attribute 'is_working_link' on <module '__main__' (built-in)>\n"
     ]
    },
    {
     "ename": "BrokenProcessPool",
     "evalue": "A process in the process pool was terminated abruptly while the future was running or pending.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mBrokenProcessPool\u001b[0m                         Traceback (most recent call last)",
      "File \u001b[0;32m<timed eval>:1\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
      "Input \u001b[0;32mIn [36]\u001b[0m, in \u001b[0;36mmultiprocess_test\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mmultiprocess_test\u001b[39m():\n\u001b[1;32m      2\u001b[0m     test_links, test_broken, test_ok \u001b[38;5;241m=\u001b[39m prep_test_links(full_links)\n\u001b[0;32m----> 3\u001b[0m     broken_count, ok_count \u001b[38;5;241m=\u001b[39m \u001b[43mmultiprocess_check_links\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtest_links\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      4\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m test_broken \u001b[38;5;241m==\u001b[39m broken_count\n\u001b[1;32m      5\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m test_ok \u001b[38;5;241m==\u001b[39m ok_count\n",
      "Input \u001b[0;32mIn [35]\u001b[0m, in \u001b[0;36mmultiprocess_check_links\u001b[0;34m(links)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mmultiprocess_check_links\u001b[39m(links):\n\u001b[1;32m      2\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m ProcessPoolExecutor(max_workers\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m2\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m executor:\n\u001b[0;32m----> 3\u001b[0m         ok_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msum\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmap\u001b[49m\u001b[43m(\u001b[49m\u001b[43mis_working_link\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlinks\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      4\u001b[0m     broken_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlen\u001b[39m(links) \u001b[38;5;241m-\u001b[39m ok_count\n\u001b[1;32m      6\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m broken_count, ok_count\n",
      "File \u001b[0;32m/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/process.py:484\u001b[0m, in \u001b[0;36m_chain_from_iterable_of_lists\u001b[0;34m(iterable)\u001b[0m\n\u001b[1;32m    478\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_chain_from_iterable_of_lists\u001b[39m(iterable):\n\u001b[1;32m    479\u001b[0m     \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    480\u001b[0m \u001b[38;5;124;03m    Specialized implementation of itertools.chain.from_iterable.\u001b[39;00m\n\u001b[1;32m    481\u001b[0m \u001b[38;5;124;03m    Each item in *iterable* should be a list.  This function is\u001b[39;00m\n\u001b[1;32m    482\u001b[0m \u001b[38;5;124;03m    careful not to keep references to yielded objects.\u001b[39;00m\n\u001b[1;32m    483\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 484\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m element \u001b[38;5;129;01min\u001b[39;00m iterable:\n\u001b[1;32m    485\u001b[0m         element\u001b[38;5;241m.\u001b[39mreverse()\n\u001b[1;32m    486\u001b[0m         \u001b[38;5;28;01mwhile\u001b[39;00m element:\n",
      "File \u001b[0;32m/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py:619\u001b[0m, in \u001b[0;36mExecutor.map.<locals>.result_iterator\u001b[0;34m()\u001b[0m\n\u001b[1;32m    616\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m fs:\n\u001b[1;32m    617\u001b[0m     \u001b[38;5;66;03m# Careful not to keep a reference to the popped future\u001b[39;00m\n\u001b[1;32m    618\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 619\u001b[0m         \u001b[38;5;28;01myield\u001b[39;00m \u001b[43mfs\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpop\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    620\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    621\u001b[0m         \u001b[38;5;28;01myield\u001b[39;00m fs\u001b[38;5;241m.\u001b[39mpop()\u001b[38;5;241m.\u001b[39mresult(end_time \u001b[38;5;241m-\u001b[39m time\u001b[38;5;241m.\u001b[39mmonotonic())\n",
      "File \u001b[0;32m/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py:444\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    442\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m CancelledError()\n\u001b[1;32m    443\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m==\u001b[39m FINISHED:\n\u001b[0;32m--> 444\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__get_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    445\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    446\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m()\n",
      "File \u001b[0;32m/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py:389\u001b[0m, in \u001b[0;36mFuture.__get_result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    387\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception:\n\u001b[1;32m    388\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 389\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[1;32m    390\u001b[0m     \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m    391\u001b[0m         \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n\u001b[1;32m    392\u001b[0m         \u001b[38;5;28mself\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "\u001b[0;31mBrokenProcessPool\u001b[0m: A process in the process pool was terminated abruptly while the future was running or pending."
     ]
    }
   ],
   "source": [
    "%time multiprocess_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04aacd0a-b19d-4be6-978a-4d8bb3bea009",
   "metadata": {},
   "source": [
    "Something is not working here, but [SO](https://stackoverflow.com/questions/62488423/brokenprocesspool-while-running-code-in-jupyter-notebook)  comes with some help. And [here](https://github.com/ipython/ipython/issues/10894) we can find more specific explanation of the issue. I'll just move multiprocessing code to a separate file as suggested and import it from there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a776d444-0a6c-417e-b1fb-5d9c775615b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from multiprocessing_check import multiprocess_check_links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "2c04f9ce-4a64-4948-9518-e4fa0f82c867",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiprocess_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = multiprocess_check_links(n=2, links=test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "5de5392a-3043-4829-80fd-da54f93b33b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 84.9 ms, sys: 27.1 ms, total: 112 ms\n",
      "Wall time: 14.7 s\n"
     ]
    }
   ],
   "source": [
    "%time multiprocess_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5509590a-f00f-4d71-aa92-0ef7956d91da",
   "metadata": {},
   "source": [
    "Again, using 2 processes results in 2 times faster execution in comaprison to sequential check. Let's check what is the optimal number of process to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "53b196cb-7f87-403d-8077-a6e72162901c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiprocess_test_performance():\n",
    "    \n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    \n",
    "    num = []\n",
    "    times = []\n",
    "    for n in (2**n for n in range(1, 7)):\n",
    "        start = time.time()\n",
    "        broken_count, ok_count = multiprocess_check_links(n, test_links)\n",
    "        assert test_broken == broken_count\n",
    "        assert test_ok == ok_count\n",
    "        end = time.time()\n",
    "        \n",
    "        num.append(n)\n",
    "        times.append(end-start)\n",
    "    return num, times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "482a1d15-aa29-4c49-8602-d1a06268e4d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "num, times = multiprocess_test_performance()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "75eedba2-269c-4a3b-b51c-fa28e2d1ce94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAOyklEQVR4nO3db2xdd33H8fdnSRhuYaSsXtckLek2ZIQQNMjaYCC2USBZh6APeEA1prJVyhO0lQmlaoY0tGebMvFH2sQUQSnSqm5aCQFVG6ErIDSJlTlNS9KGUDYKxGmJK+YxMWuk4bsHPu4c16n/3Gvf+3PeL+nq3vM7x/d8f8nJxye/8y9VhSSpPT8z6AIkSatjgEtSowxwSWqUAS5JjTLAJalRm9dzZVdeeWXt3LlzPVcpSc07evTo01U1urB9XQN8586dTExMrOcqJal5Sb67WLtDKJLUKANckhplgEtSowxwSWqUAS5JjVrXs1BW4/CxSQ4cOcWZ6Rm2bR1h3+4xbtq1fdBlSdLADXWAHz42yf5Dx5k5dx6AyekZ9h86DmCIS7rkDfUQyoEjp54N7zkz585z4MipAVUkScNjqAP8zPTMitol6VIy1AG+bevIitol6VIy1AG+b/cYI1s2XdA2smUT+3aPDagiSRoeQ30Qc+5ApWehSNJzDXWAw2yIG9iS9FxDPYQiSbo4A1ySGmWAS1KjlgzwJHcmOZvkxCLzPpCkkly5NuVJki5mOXvgdwF7FjYmuQZ4G/C9PtckSVqGJQO8qr4K/HCRWR8Bbgeq30VJkpa2qjHwJO8EJqvqkT7XI0laphWfB57kMuBPmB0+Wc7ye4G9ANdee+1KVydJuojV7IH/MnAd8EiSJ4AdwENJfnGxhavqYFWNV9X46Ojo6iuVJF1gxXvgVXUc+IW56S7Ex6vq6T7WJUlawnJOI7wH+BowluR0klvXvixJ0lKW3AOvqpuXmL+zb9VIkpbNKzElqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjlgzwJHcmOZvkxLy2A0m+meQbST6bZOuaVilJeo7l7IHfBexZ0HY/8KqqejXwLWB/n+uSJC1hyQCvqq8CP1zQ9sWqeqab/FdgxxrUJkl6Hv0YA/8D4J8uNjPJ3iQTSSampqb6sDpJEvQY4Ek+CDwD3H2xZarqYFWNV9X46OhoL6uTJM2zebU/mOS9wNuBG6qq+laRJGlZVhXgSfYAtwO/UVX/09+SJEnLsZzTCO8BvgaMJTmd5Fbgr4AXA/cneTjJ36xxnZKkBZbcA6+qmxdp/uQa1CJJWgGvxJSkRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElq1JIBnuTOJGeTnJjX9tIk9yd5vHu/Ym3LlCQttJw98LuAPQva7gAeqKqXAw9005KkdbRkgFfVV4EfLmh+J/Dp7vOngZv6W5YkaSmrHQO/qqqe7D4/BVx1sQWT7E0ykWRiampqlauTJC3U80HMqiqgnmf+waoar6rx0dHRXlcnSeqsNsB/kORqgO79bP9KkiQtx2oD/PPALd3nW4DP9accSdJyLec0wnuArwFjSU4nuRX4c+CtSR4H3tJNS5LW0ealFqiqmy8y64Y+1yJJWgGvxJSkRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqVE8BnuSPkzya5ESSe5K8sF+FSZKe36oDPMl24I+A8ap6FbAJeHe/CpMkPb9eh1A2AyNJNgOXAWd6L0mStByrDvCqmgT+Evge8CTwX1X1xYXLJdmbZCLJxNTU1OorlSRdoJchlCuAdwLXAduAy5O8Z+FyVXWwqsaranx0dHT1lUqSLtDLEMpbgO9U1VRVnQMOAb/en7IkSUvpJcC/B7wuyWVJAtwAnOxPWZKkpfQyBv4gcC/wEHC8+66DfapLkrSEzb38cFV9CPhQn2qRJK1ATwE+TA4fm+TAkVOcmZ5h29YR9u0e46Zd2wddliStmQ0R4IePTbL/0HFmzp0HYHJ6hv2HjgMY4pI2rA1xL5QDR049G95zZs6d58CRUwOqSJLW3oYI8DPTMytql6SNYEME+LatIytql6SNYEME+L7dY4xs2XRB28iWTezbPTagiiRp7W2Ig5hzByo9C0XSpWRDBDjMhriBLelSsiGGUCTpUmSAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRPQV4kq1J7k3yzSQnk7y+X4VJkp5fr7eT/Rjwhap6V5IXAJf1oSZJ0jKsOsCTvAR4E/BegKr6CfCT/pQlSVpKL0Mo1wFTwKeSHEvyiSSXL1woyd4kE0kmpqamelidJGm+XgJ8M/Ba4ONVtQv4MXDHwoWq6mBVjVfV+OjoaA+rkyTN10uAnwZOV9WD3fS9zAa6JGkdrDrAq+op4PtJ5h79fgPwWF+qkiQtqdezUP4QuLs7A+U/gN/vvSRJ0nL0FOBV9TAw3p9SJEkr4ZWYktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY3qOcCTbEpyLMl9/ShIkrQ8/dgDvw042YfvkSStQE8BnmQH8DvAJ/pTjiRpuXrdA/8ocDvw04stkGRvkokkE1NTUz2uTpI0Z9UBnuTtwNmqOvp8y1XVwaoar6rx0dHR1a5OkrTA5h5+9g3AO5LcCLwQ+Lkkf1tV7+lPaYN1+NgkB46c4sz0DNu2jrBv9xg37do+6LIk6Vmr3gOvqv1VtaOqdgLvBr60kcJ7/6HjTE7PUMDk9Az7Dx3n8LHJQZcmSc/yPPBFHDhyiplz5y9omzl3ngNHTg2oIkl6rl6GUJ5VVV8BvtKP7xoGZ6ZnVtQuSYPgHvgitm0dWVG7JA2CAb6IfbvHGNmy6YK2kS2b2Ld7bEAVSdJz9WUIZaOZO9vEs1AkDTMD/CJu2rXdwJY01BxCkaRGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjvJlVg3xepyQwwJsz97zOuUe+zT2vEzDEpUuMQyiN8XmdkuYY4I3xeZ2S5hjgjfF5nZLmrDrAk1yT5MtJHkvyaJLb+lmYFufzOiXN6eUg5jPAB6rqoSQvBo4mub+qHutTbVqEz+uUNGfVAV5VTwJPdp//O8lJYDtggK8xn9cpCfp0GmGSncAu4MFF5u0F9gJce+21/VidJDVjLa/b6PkgZpIXAZ8B3l9VP1o4v6oOVtV4VY2Pjo72ujpJasbcdRuT0zMU/3/dxuFjk335/p4CPMkWZsP77qo61JeKJGmDWOvrNlY9hJIkwCeBk1X14b5UI60hb0Gg9bbW1230sgf+BuD3gDcnebh73diXqqQ+W+v/ykqLWevrNlYd4FX1L1WVqnp1VV3fvf6xL1VJfeYtCDQIa33dhjez0iXBWxBoENb6ug0DXJeEbVtHmFwkrL0FgdbaWl634b1QdEnwFgTaiNwD1yXBWxBoIzLAdcnwFgTaaBxCkaRGGeCS1CgDXJIaZYBLUqMMcElqVKpq/VaWTAHfXdB8JfD0uhWxNjZCH2Bj9MM+DAf70F8vq6rn3I97XQN8MUkmqmp8oEX0aCP0ATZGP+zDcLAP68MhFElqlAEuSY0ahgA/OOgC+mAj9AE2Rj/sw3CwD+tg4GPgkqTVGYY9cEnSKhjgktSogQZ4kj1JTiX5dpI7BlnLciW5M8nZJCfmtb00yf1JHu/erxhkjUtJck2SLyd5LMmjSW7r2pvpR5IXJvl6kke6PvxZ135dkge7bervk7xg0LUuJcmmJMeS3NdNN9WHJE8kOd49F3eia2tmWwJIsjXJvUm+meRkkte30IeBBXiSTcBfA78NvBK4OckrB1XPCtwF7FnQdgfwQFW9HHigmx5mzwAfqKpXAq8D3tf92bfUj/8F3lxVrwGuB/YkeR3wF8BHqupXgP8Ebh1cict2G3By3nSLffit7rm4c+dNt7QtAXwM+EJVvQJ4DbN/H8Pfh6oayAt4PXBk3vR+YP+g6llh7TuBE/OmTwFXd5+vBk4NusYV9udzwFtb7QdwGfAQ8GvMXjm3uWu/YBsbxhewg9lweDNwH5AG+/AEcOWCtma2JeAlwHfoTupoqQ+DHELZDnx/3vTprq1FV1XVk93np4CrBlnMSiTZCewCHqSxfnRDDw8DZ4H7gX8HpqvqmW6RFrapjwK3Az/tpn+e9vpQwBeTHE2yt2traVu6DpgCPtUNZX0iyeU00AcPYvZZzf66buLczCQvAj4DvL+qfjR/Xgv9qKrzVXU9s3uxvwq8YrAVrUyStwNnq+rooGvp0Rur6rXMDoe+L8mb5s9sYFvaDLwW+HhV7QJ+zILhkmHtwyADfBK4Zt70jq6tRT9IcjVA9352wPUsKckWZsP77qo61DU31w+AqpoGvszscMPWJHOPChz2beoNwDuSPAH8HbPDKB+jrT5QVZPd+1ngs8z+Mm1pWzoNnK6qB7vpe5kN9KHvwyAD/N+Al3dH3F8AvBv4/ADr6cXngVu6z7cwO6Y8tJIE+CRwsqo+PG9WM/1IMppka/d5hNkx/JPMBvm7usWGug9Vtb+qdlTVTma3/y9V1e/SUB+SXJ7kxXOfgbcBJ2hoW6qqp4DvJxnrmm4AHqOFPgz44MGNwLeYHbv84KAPCCyz5nuAJ4FzzP7mvpXZccsHgMeBfwZeOug6l+jDG5n97+A3gIe7140t9QN4NXCs68MJ4E+79l8Cvg58G/gH4GcHXesy+/ObwH2t9aGr9ZHu9ejcv+OWtqWu3uuBiW57Ogxc0UIfvJRekhrlQUxJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhr1fxKeqLQV06cBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(num, times)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22103eac-ec4d-4a81-9125-402b9648a69c",
   "metadata": {},
   "source": [
    "In contrast to using multiple threads, with multiple process we can clearly see that the overhead for creating > 30 processes increases total time of execution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb90fd11-705c-459c-8cf8-fb282c9e90d5",
   "metadata": {},
   "source": [
    "Final test for multiprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "bbe090c4-9cdf-4036-a82b-f7dc0c8d6640",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiprocess_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = multiprocess_check_links(n=16, links=test_links)\n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "608d6b68-e363-4aea-a51c-774b69c7d1f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 47.5 ms, sys: 152 ms, total: 199 ms\n",
      "Wall time: 2.12 s\n"
     ]
    }
   ],
   "source": [
    "%time multiprocess_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e994985e-14fb-43b3-96b5-9a8b4775ef6d",
   "metadata": {},
   "source": [
    "This result is surprisingly similar to the execution time when we used multithreading."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf58bbb4-3618-4bc4-ba75-0670e159c77e",
   "metadata": {},
   "source": [
    "As we know from \"The Zen of Python\" (`import this`) - \"There should be one-- and preferably only one --obvious way to do it.\"  \n",
    "How do we choose between multithreading vs multiprocessing given that they have very similar performance for our task?\n",
    "The second most common concern for programs is the memory consumption and we can use it to get more insight about preferable choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "60a0d8ee-615a-46f8-aed6-92aee05bd68c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext memory_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b5a67656-860b-4e08-87a4-9b125cc8c278",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 206.97 MiB, increment: 1.00 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit thread_test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8e179dc0-91d0-4ec4-94ae-65948a8f5c0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 206.50 MiB, increment: 0.03 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit multiprocess_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fbe2684-eb92-4958-a65e-4be8edf9b993",
   "metadata": {},
   "source": [
    "Wow there is surprinsingly little difference between multithreading and multiprocessing for our task. I'm not sure why it is like that, but I think that it might be related to running these profiling experiments in Jupyter Notebook. Let's move final time/memory profiling to plain files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8c679d-ea50-4134-a92d-2808c870a42d",
   "metadata": {},
   "source": [
    "## AsyncIO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82471e05-be3a-412e-b67d-f4d420e0c07a",
   "metadata": {},
   "source": [
    "Threads and processes seem to be more common and better understood, so let's start with the question - what is async IO?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f4a970-43db-4d6f-a14b-f8433de1b323",
   "metadata": {},
   "source": [
    "Async IO is a style of concurrent programming. It's single-threaded and single-process with concurrency happening at the function level. Async IO functions are called coroutines and are a specialized version of a Python generator functions. A coroutine is a function that can suspend its execution before reaching return, and it can indirectly pass control to another coroutine for some time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7a920e-2463-4bb9-be64-1ceae9ff33c4",
   "metadata": {},
   "source": [
    "### AsyncIO Hello World"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "4b31bb60-337a-482e-b065-d8e68f836fa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "async def hello_world():\n",
    "    print('Hello')\n",
    "    await asyncio.sleep(1)\n",
    "    print('World')\n",
    "    \n",
    "async def main():\n",
    "    await asyncio.gather(*(hello_world() for _ in range(10))) # see later comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "bf9652e5-8d27-4081-ba3d-2d823d53ea92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "Hello\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n",
      "World\n"
     ]
    }
   ],
   "source": [
    "s = time.time()\n",
    "await main()\n",
    "e = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4ae7a37e-c7c6-4d79-980e-e9020d11e98c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 1.01 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f'Took {e - s:.2f} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d91c52c-b889-443a-b1bf-94b3a1b6acd8",
   "metadata": {},
   "source": [
    "Execution takes 1 second, because on the one hand sleeping for 1 second is performed, but coroutine doesn't block execution of another coroutine, so effectively they all are waiting 1 second at the same time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57477129-a1b0-4789-bd22-6bce7de1b9ed",
   "metadata": {},
   "source": [
    "The `main` function with its\n",
    "```python\n",
    "await asyncio.gather(*(hello_world() for _ in range(10)))\n",
    "```\n",
    "might be a bit too ambitious to call it \"Hello World\" example, so let's unpack what's going on there."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a4f1ff-3529-4f3b-8020-0a6e43deed14",
   "metadata": {},
   "source": [
    "1. `await` keyword is used to actually run coroutines. Once we define async function, we can use `await` to run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "e549daee-362a-4543-bd02-cbec5afd524d",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def hey():\n",
    "    return 'hey'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "3133044e-c95e-47b8-9d3e-f35e47abf587",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<coroutine object hey at 0x10a130bc0>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hey()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51435ec6-1fcf-4d88-a2ef-7f2a73b8b4ef",
   "metadata": {},
   "source": [
    "Calling async function normally does not return any result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b12ed6fb-baa5-4d30-96d9-95715d12f36c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hey'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await hey()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5098f7c2-8c7b-4d74-8ace-428b9b72cb4f",
   "metadata": {},
   "source": [
    "`await` is a keyword we can use for running async function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b847b2c-2644-4c37-9cfd-47fdf8e9f344",
   "metadata": {},
   "source": [
    "2. `asyncio.gather` - is used to run sequence of coroutines concurrently.  \n",
    "We usually multiplpe define async functions and then run them concurrently and `gather` is exactly a mechanism used for requesting concurrent execution of async functions. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "91980053-ad19-4057-acef-ae89dd47f556",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hey', 'hey', 'hey']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await asyncio.gather(hey(), hey(), hey())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e0b2ca-4372-4bdd-b87e-926dac0a984d",
   "metadata": {},
   "source": [
    "3. `*(hello_world() for _ in range(10))` - I want to run 10 coroutines `hello_world`, but I obviously don't want to type it 10 times. Instead, I'm using generator expression `(hello_world() for _ in range(10)` that yields 10 coroutines. However, `gather` function accepts each coroutine as a separate argument, so I'm unpacking my generator with `*`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "b014a645-fa2a-4eca-9e2e-687a6824b1a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object <genexpr> at 0x1348ec120>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(hello_world for _ in range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "84b3eb57-239e-44af-8bbf-7fd464f851fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790> <function hello_world at 0x134971790>\n"
     ]
    }
   ],
   "source": [
    "print(*(hello_world for _ in range(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81fcc653-1761-4008-9a21-3e7e4b276bdb",
   "metadata": {},
   "source": [
    "### AsyncIO requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "072d7b10-eda3-47da-b262-aa0ea776b437",
   "metadata": {},
   "source": [
    "`requests` library I was using for all the previous examples makes sequetial network calls that we were able to speed up by spreading them over multiple threads or process. This is not going to work with Async IO model, and to make network calls in coroutines we need to use `aiohttp` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "ea3f815a-51b0-4154-a5b5-919cb0fca1d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import aiohttp\n",
    "from aiohttp import client_exceptions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d843e2-83e8-4efa-85c4-9150bf762acd",
   "metadata": {},
   "source": [
    "Let's start with async `check_link`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "22ac57fd-e4c2-426f-8dd0-780bb9b67f7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def check_link(session, link):\n",
    "    try:\n",
    "        async with session.get(link) as response:\n",
    "            return response.status == 200\n",
    "    except client_exceptions.ClientConnectionError as e:\n",
    "        return False\n",
    "    else:\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "21ce5ae3-e959-4843-b70b-f3cf247b5fc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Unclosed client session\n",
      "client_session: <aiohttp.client.ClientSession object at 0x134bc0610>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await check_link(aiohttp.ClientSession(), 'https://en.wikipedia.org//wiki/Japan')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b08127a-5d9c-4b9a-b7bd-d61caadfbfd1",
   "metadata": {},
   "source": [
    "Seems to be working fine for one exmaple, so let's prepare full test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "c9ad2999-ed59-47f7-a6e6-db0937960441",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def async_check_links(links):\n",
    "    async with aiohttp.ClientSession() as session:\n",
    "        tasks = []\n",
    "        for link in links:\n",
    "            tasks.append(check_link(session, link))\n",
    "            \n",
    "        results = await asyncio.gather(*tasks)\n",
    "\n",
    "    ok_count = sum(result for result in results)    \n",
    "    broken_count = len(links) - ok_count\n",
    "    \n",
    "    return broken_count, ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "352de4f0-888a-4f74-8ffe-a77161f69bb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def async_test():\n",
    "    test_links, test_broken, test_ok = prep_test_links(full_links)\n",
    "    broken_count, ok_count = await async_check_links(test_links)\n",
    "    \n",
    "    assert test_broken == broken_count\n",
    "    assert test_ok == ok_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "9dfce5f6-a31e-453e-b64c-f93416c45778",
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "'await' outside function (<timed eval>, line 1)",
     "output_type": "error",
     "traceback": [
      "Traceback \u001b[0;36m(most recent call last)\u001b[0m:\n",
      "  File \u001b[1;32m~/code/link-checker/.venv/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3369\u001b[0m in \u001b[1;35mrun_code\u001b[0m\n    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  Input \u001b[1;32mIn [63]\u001b[0m in \u001b[1;35m<cell line: 1>\u001b[0m\n    get_ipython().run_line_magic('time', 'await async_test()')\n",
      "  File \u001b[1;32m~/code/link-checker/.venv/lib/python3.8/site-packages/IPython/core/interactiveshell.py:2294\u001b[0m in \u001b[1;35mrun_line_magic\u001b[0m\n    result = fn(*args, **kwargs)\n",
      "  File \u001b[1;32m~/code/link-checker/.venv/lib/python3.8/site-packages/IPython/core/magics/execution.py:1297\u001b[0m in \u001b[1;35mtime\u001b[0m\n    code = self.shell.compile(expr_ast, source, mode)\n",
      "\u001b[0;36m  File \u001b[0;32m/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/codeop.py:143\u001b[0;36m in \u001b[0;35m__call__\u001b[0;36m\u001b[0m\n\u001b[0;31m    codeob = compile(source, filename, symbol, self.flags, 1)\u001b[0m\n",
      "\u001b[0;36m  File \u001b[0;32m<timed eval>:1\u001b[0;36m\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m 'await' outside function\n"
     ]
    }
   ],
   "source": [
    "%time await async_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b589b40a-04e5-49f1-ac5a-9a83c5d12cea",
   "metadata": {},
   "source": [
    "Jupyter doesn't seem to work well with async code. Let's run one simple test here and move further profiling to files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "f5bebb06-2ab2-4c05-bba6-d96f155d5a4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 0.80 seconds\n"
     ]
    }
   ],
   "source": [
    "s = time.time()\n",
    "await async_test()\n",
    "e = time.time()\n",
    "print(f'Took {e - s:.2f} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "595651cb-a0a4-44d3-9b58-20e70af04ad5",
   "metadata": {},
   "source": [
    "Even though AsyncIO requires special syntax and is not as easy to write as the other methods, it provides the fastest execution for network requests. Let's move on to a more detailed profiling in files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}