company.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping Dynamic Website using Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll walk you through the process of using Python and Selenium to scrape company data from the Affiliate World Conferences Europe Exhibitors website  ( https://affiliateworldconferences.com/europe/exhibitors/ ) . We'll cover how to automate web scraping, extract relevant information, and organize the data into a structured format.  \n",
    "We'll scrape the company's name, type, location, description, and website URL that are not associated with Facebook or Twitter.\n",
    "Consider the following picture, we need to scrape the that information:  \n",
    "![company data](companyData3.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites  \n",
    "Before you start, make sure you have the following installed:  \n",
    "- Python (3.x recommended)  \n",
    "- Selenium library  \n",
    "- Pandas library  \n",
    "- ChromeDriver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Web Driver  \n",
    "The Chrome WebDriver enables you to open web pages, interact with elements on the page (clicking buttons, filling out forms, etc.), navigate between pages, and retrieve information from the page's source code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the required libraries\n",
    "from selenium import webdriver\n",
    "import os\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "chromedriverPath = r'C:\\Users\\driver\\chromedriver.exe'\n",
    "# Create a Service object with the path to chromedriver\n",
    "chromeService = webdriver.chrome.service.Service(chromedriverPath)\n",
    "# Start the service\n",
    "chromeService.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the WebDriver using the Service object\n",
    "driver = webdriver.Chrome(service=chromeService)\n",
    "driver.get('https://affiliateworldconferences.com/europe/exhibitors/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Toshiba\\AppData\\Local\\Temp\\ipykernel_15484\\514897752.py:13: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  childElement = driver.find_element_by_css_selector(childSelector)\n"
     ]
    }
   ],
   "source": [
    "# CSS selector of the parent element containing child elements\n",
    "parentSelector = '#__next > main > section > div > section > ul.styles__List-sc-izf6gv-19.bsFaZO'\n",
    "#__next > main > section > div > section > ul.styles__List-sc-izf6gv-19.bsFaZO > li:nth-child(1)\n",
    "#__next > main > section > div > section > ul.styles__List-sc-izf6gv-19.bsFaZO > li:nth-child(2)\n",
    "#__next > main > section > div > section > ul.styles__List-sc-izf6gv-19.bsFaZO > li:nth-child(202)\n",
    "companiesList = []\n",
    "for i in range(155, 163):\n",
    "        if i == 160:\n",
    "            continue  # Skip iteration for child number 160\n",
    "        try:\n",
    "            childSelector = f\"{parentSelector} > li:nth-child({i})\"\n",
    "            # Click on the child element to open the pop-up\n",
    "            childElement = driver.find_element_by_css_selector(childSelector)\n",
    "            driver.execute_script(\"arguments[0].click();\", childElement)\n",
    "\n",
    "            # Wait for the pop-up to load (if needed)\n",
    "            wait = WebDriverWait(driver, 10)\n",
    "            # Select the required elements within the pop-up\n",
    "            h3Elements = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.body h3')))\n",
    "            pElements = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.body p')))\n",
    "            aElements = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.body a')))\n",
    "\n",
    "            h3Texts = [h3Element.text for h3Element in h3Elements]\n",
    "            \n",
    "            if pElements:\n",
    "\n",
    "                typeAndLocation = [pElements[0].text for pElement in pElements]\n",
    "\n",
    "                # Split the specificText into Type and Location\n",
    "                for types in typeAndLocation:\n",
    "                    companyType, location = map(str.strip, types.split('|', 1))\n",
    "\n",
    "                # ppText = [pElements[2].text]\n",
    "            \n",
    "                \n",
    "                # Scrape pElements[2] if available for the current child\n",
    "                uniqueTexts = []      \n",
    "                specificTexts = []\n",
    "                \n",
    "                # Scrape pElements[1] if available for the current child\n",
    "                if len(pElements) == 2:\n",
    "                    specificText1 = pElements[1].text\n",
    "                    if specificText1:\n",
    "                        specificTexts.append(specificText1)\n",
    "\n",
    "                if len(pElements) >= 3:\n",
    "                    specificText1 = pElements[2].text\n",
    "                    if specificText1:\n",
    "                        specificTexts.append(specificText1)\n",
    "\n",
    "            # Scrape pElements[3] if available for the current child\n",
    "                if len(pElements) >= 4:\n",
    "                    specificText2 = pElements[3].text\n",
    "                    if specificText2:\n",
    "                        specificTexts.append(specificText2)\n",
    "\n",
    "        # Combine the texts for the current child if both [2] and [3] are available\n",
    "                combinedText = \" AND \".join(specificTexts)\n",
    "        \n",
    "        # Append the combined text to the uniqueTexts list if it's not empty\n",
    "                if combinedText and combinedText not in uniqueTexts:\n",
    "                    uniqueTexts.append(combinedText)\n",
    "            else:\n",
    "                # Handle the case where pElements is empty (no elements found)\n",
    "                print(f\"No 'p' elements found in child {i}\")\n",
    "\n",
    "            websiteLink = None\n",
    "            for aElement in aElements:\n",
    "                hrefValue = aElement.get_attribute('href')\n",
    "                if hrefValue and 'facebook' not in hrefValue.lower() and 'twitter' not in hrefValue.lower():\n",
    "                    websiteLink = hrefValue\n",
    "                    break\n",
    "            # creat diction to store the element in companiesList\n",
    "            container ={\n",
    "                'name': h3Texts,\n",
    "                'type': companyType,\n",
    "                'location': location,\n",
    "                'description': uniqueTexts,\n",
    "                'url': websiteLink\n",
    "                }\n",
    "            companiesList.append(container)\n",
    "            # print('saving: ', container['name'])\n",
    "\n",
    "        except Exception as e:\n",
    "            print(f\"Error while scraping child {i}: {str(e)}\")\n",
    "            continue\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': ['PureVPN - White Label Solution'],\n",
       "  'type': 'Service Provider',\n",
       "  'location': 'Virgin Islands, British',\n",
       "  'description': ['PureWL enables businesses, enterprises, & super affiliates with a revolutionary white label VPN solution. Our solution is built from the ground up with extensive knowledge and experience in a constantly evolving cybersecurity landscape.'],\n",
       "  'url': 'https://www.purewl.com/'},\n",
       " {'name': ['Quantox Technology'],\n",
       "  'type': 'Service Provider',\n",
       "  'location': 'Serbia',\n",
       "  'description': ['Quantox Technology is an international company specializing in software development, IT business solutions, and consulting. It has over 500 tech experts across 7 countries and is known as the fastest-growing company in Eastern Europe.'],\n",
       "  'url': 'https://www.quantox.com/'},\n",
       " {'name': ['REBLL Network'],\n",
       "  'type': 'Network',\n",
       "  'location': 'Netherlands',\n",
       "  'description': ['REBLL has a solid portfolio of in-house dating offers. Next to that, we partner up with the best in the affiliate industry, to bring you offers that outperform their competition.'],\n",
       "  'url': 'https://rebll.com/'},\n",
       " {'name': ['Revolution Force'],\n",
       "  'type': 'Network',\n",
       "  'location': 'United States',\n",
       "  'description': ['Revolution Force is a performance affiliate network specializing in the dating vertical.'],\n",
       "  'url': 'https://revolutionforce.com/'},\n",
       " {'name': ['RichAds'],\n",
       "  'type': 'Traffic Source',\n",
       "  'location': 'Cyprus',\n",
       "  'description': ['RichAds is a self-serve advertising platform where scale meets performance.Our advertising formats: push, pops, in-page, calendar, direct click. RichAds offers 5B impressions daily in 200+ countries worldwide.'],\n",
       "  'url': 'https://richads.com/?utm_source=awe_23'},\n",
       " {'name': ['RollerAds'],\n",
       "  'type': 'Network',\n",
       "  'location': 'United States',\n",
       "  'description': ['RollerAds is a high-performance advertising network that exclusively monetizes all the SendPulse push notifications and tons of various direct traffic.'],\n",
       "  'url': 'https://rollerads.com/'},\n",
       " {'name': ['RoundSky'],\n",
       "  'type': 'Advertiser',\n",
       "  'location': 'United States',\n",
       "  'description': ['Round Sky offers direct USA loan offers. We own all our offers direct and they convert over 80% and pay up to $265 with EPCs averaging over $1. We also have API/XML posting, payday loans, installment loans, and more.'],\n",
       "  'url': 'https://www.roundsky.com/'}]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "companiesList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame from the 'data' list\n",
    "df = pd.DataFrame(companiesList)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>location</th>\n",
       "      <th>description</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[PureVPN - White Label Solution]</td>\n",
       "      <td>Service Provider</td>\n",
       "      <td>Virgin Islands, British</td>\n",
       "      <td>[PureWL enables businesses, enterprises, &amp; super affiliates with a revolutionary white label VPN solution. Our solution is built from the ground up with extensive knowledge and experience in a constantly evolving cybersecurity landscape.]</td>\n",
       "      <td>https://www.purewl.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Quantox Technology]</td>\n",
       "      <td>Service Provider</td>\n",
       "      <td>Serbia</td>\n",
       "      <td>[Quantox Technology is an international company specializing in software development, IT business solutions, and consulting. It has over 500 tech experts across 7 countries and is known as the fastest-growing company in Eastern Europe.]</td>\n",
       "      <td>https://www.quantox.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[REBLL Network]</td>\n",
       "      <td>Network</td>\n",
       "      <td>Netherlands</td>\n",
       "      <td>[REBLL has a solid portfolio of in-house dating offers. Next to that, we partner up with the best in the affiliate industry, to bring you offers that outperform their competition.]</td>\n",
       "      <td>https://rebll.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Revolution Force]</td>\n",
       "      <td>Network</td>\n",
       "      <td>United States</td>\n",
       "      <td>[Revolution Force is a performance affiliate network specializing in the dating vertical.]</td>\n",
       "      <td>https://revolutionforce.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[RichAds]</td>\n",
       "      <td>Traffic Source</td>\n",
       "      <td>Cyprus</td>\n",
       "      <td>[RichAds is a self-serve advertising platform where scale meets performance.Our advertising formats: push, pops, in-page, calendar, direct click. RichAds offers 5B impressions daily in 200+ countries worldwide.]</td>\n",
       "      <td>https://richads.com/?utm_source=awe_23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[RollerAds]</td>\n",
       "      <td>Network</td>\n",
       "      <td>United States</td>\n",
       "      <td>[RollerAds is a high-performance advertising network that exclusively monetizes all the SendPulse push notifications and tons of various direct traffic.]</td>\n",
       "      <td>https://rollerads.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[RoundSky]</td>\n",
       "      <td>Advertiser</td>\n",
       "      <td>United States</td>\n",
       "      <td>[Round Sky offers direct USA loan offers. We own all our offers direct and they convert over 80% and pay up to $265 with EPCs averaging over $1. We also have API/XML posting, payday loans, installment loans, and more.]</td>\n",
       "      <td>https://www.roundsky.com/</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               name              type  \\\n",
       "0  [PureVPN - White Label Solution]  Service Provider   \n",
       "1              [Quantox Technology]  Service Provider   \n",
       "2                   [REBLL Network]           Network   \n",
       "3                [Revolution Force]           Network   \n",
       "4                         [RichAds]    Traffic Source   \n",
       "5                       [RollerAds]           Network   \n",
       "6                        [RoundSky]        Advertiser   \n",
       "\n",
       "                  location  \\\n",
       "0  Virgin Islands, British   \n",
       "1                   Serbia   \n",
       "2              Netherlands   \n",
       "3            United States   \n",
       "4                   Cyprus   \n",
       "5            United States   \n",
       "6            United States   \n",
       "\n",
       "                                                                                                                                                                                                                                      description  \\\n",
       "0  [PureWL enables businesses, enterprises, & super affiliates with a revolutionary white label VPN solution. Our solution is built from the ground up with extensive knowledge and experience in a constantly evolving cybersecurity landscape.]   \n",
       "1    [Quantox Technology is an international company specializing in software development, IT business solutions, and consulting. It has over 500 tech experts across 7 countries and is known as the fastest-growing company in Eastern Europe.]   \n",
       "2                                                            [REBLL has a solid portfolio of in-house dating offers. Next to that, we partner up with the best in the affiliate industry, to bring you offers that outperform their competition.]   \n",
       "3                                                                                                                                                      [Revolution Force is a performance affiliate network specializing in the dating vertical.]   \n",
       "4                             [RichAds is a self-serve advertising platform where scale meets performance.Our advertising formats: push, pops, in-page, calendar, direct click. RichAds offers 5B impressions daily in 200+ countries worldwide.]   \n",
       "5                                                                                       [RollerAds is a high-performance advertising network that exclusively monetizes all the SendPulse push notifications and tons of various direct traffic.]   \n",
       "6                      [Round Sky offers direct USA loan offers. We own all our offers direct and they convert over 80% and pay up to $265 with EPCs averaging over $1. We also have API/XML posting, payday loans, installment loans, and more.]   \n",
       "\n",
       "                                      url  \n",
       "0                 https://www.purewl.com/  \n",
       "1                https://www.quantox.com/  \n",
       "2                      https://rebll.com/  \n",
       "3            https://revolutionforce.com/  \n",
       "4  https://richads.com/?utm_source=awe_23  \n",
       "5                  https://rollerads.com/  \n",
       "6               https://www.roundsky.com/  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set max column width outh truncation\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to remove the square brackets from the name and description columns\n",
    "df['name'] = df['name'].apply(lambda x: ', '.join(x).replace('[', '').replace(']', ''))\n",
    "df['description'] = df['description'].apply(lambda x: ', '.join(x).replace('[', '').replace(']', ''))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>location</th>\n",
       "      <th>description</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PureVPN - White Label Solution</td>\n",
       "      <td>Service Provider</td>\n",
       "      <td>Virgin Islands, British</td>\n",
       "      <td>PureWL enables businesses, enterprises, &amp; super affiliates with a revolutionary white label VPN solution. Our solution is built from the ground up with extensive knowledge and experience in a constantly evolving cybersecurity landscape.</td>\n",
       "      <td>https://www.purewl.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Quantox Technology</td>\n",
       "      <td>Service Provider</td>\n",
       "      <td>Serbia</td>\n",
       "      <td>Quantox Technology is an international company specializing in software development, IT business solutions, and consulting. It has over 500 tech experts across 7 countries and is known as the fastest-growing company in Eastern Europe.</td>\n",
       "      <td>https://www.quantox.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>REBLL Network</td>\n",
       "      <td>Network</td>\n",
       "      <td>Netherlands</td>\n",
       "      <td>REBLL has a solid portfolio of in-house dating offers. Next to that, we partner up with the best in the affiliate industry, to bring you offers that outperform their competition.</td>\n",
       "      <td>https://rebll.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Revolution Force</td>\n",
       "      <td>Network</td>\n",
       "      <td>United States</td>\n",
       "      <td>Revolution Force is a performance affiliate network specializing in the dating vertical.</td>\n",
       "      <td>https://revolutionforce.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RichAds</td>\n",
       "      <td>Traffic Source</td>\n",
       "      <td>Cyprus</td>\n",
       "      <td>RichAds is a self-serve advertising platform where scale meets performance.Our advertising formats: push, pops, in-page, calendar, direct click. RichAds offers 5B impressions daily in 200+ countries worldwide.</td>\n",
       "      <td>https://richads.com/?utm_source=awe_23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             name              type                 location  \\\n",
       "0  PureVPN - White Label Solution  Service Provider  Virgin Islands, British   \n",
       "1              Quantox Technology  Service Provider                   Serbia   \n",
       "2                   REBLL Network           Network              Netherlands   \n",
       "3                Revolution Force           Network            United States   \n",
       "4                         RichAds    Traffic Source                   Cyprus   \n",
       "\n",
       "                                                                                                                                                                                                                                    description  \\\n",
       "0  PureWL enables businesses, enterprises, & super affiliates with a revolutionary white label VPN solution. Our solution is built from the ground up with extensive knowledge and experience in a constantly evolving cybersecurity landscape.   \n",
       "1    Quantox Technology is an international company specializing in software development, IT business solutions, and consulting. It has over 500 tech experts across 7 countries and is known as the fastest-growing company in Eastern Europe.   \n",
       "2                                                            REBLL has a solid portfolio of in-house dating offers. Next to that, we partner up with the best in the affiliate industry, to bring you offers that outperform their competition.   \n",
       "3                                                                                                                                                      Revolution Force is a performance affiliate network specializing in the dating vertical.   \n",
       "4                             RichAds is a self-serve advertising platform where scale meets performance.Our advertising formats: push, pops, in-page, calendar, direct click. RichAds offers 5B impressions daily in 200+ countries worldwide.   \n",
       "\n",
       "                                      url  \n",
       "0                 https://www.purewl.com/  \n",
       "1                https://www.quantox.com/  \n",
       "2                      https://rebll.com/  \n",
       "3            https://revolutionforce.com/  \n",
       "4  https://richads.com/?utm_source=awe_23  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Once 'openpyxl' is installed, you should be able to run your code and \n",
    "# save the DataFrame to an Excel file without any issues.\n",
    "# pip install openpyxl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the DataFrame to an Excel file\n",
    "df.to_excel('companiesList.xlsx', index=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}