Quakers-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network project presentation "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Santino NANINI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Content \n",
    " \n",
    "    - Title : \n",
    "    \"Quaker's social network investigation: a two-way soft complementary approach\"\n",
    "    \n",
    "    - Introduction and motivation: \n",
    "    The historical context is interessting about understanding how religious people fouded there own religious\n",
    "    order centuries ago in a \"new\" state.\n",
    "    There was an accessible, open data set on the internet and the site was also providing a certain help for \n",
    "    historical-investigations. I liked the fact that I could run the data on Gephi in parallel (because there \n",
    "    was none) to bring supplement visualizations from my own side.\n",
    "    Furtermore, I wanted to understand the code that the historians used to investigate the Quakers' social \n",
    "    network and compare it with the graphs and results that I get with Gephi.\n",
    "    \n",
    "    - Choosing the dataset, overview of available dataset, loading data:\n",
    "    Dataset is derived from the \"Oxford Dictionary of National Biography\" and from the ongoing work of the \"Six \n",
    "    Degrees of Francis Bacon\" project.\n",
    "    The data is divided in \"quakers_nodelist.csv\", a list of early modern Quakers (nodes) and the file \n",
    "    quakers_edgelist.csv\", a list of relationships between those Quakers (edges) and can be respectively found:\n",
    "    https://programminghistorian.org/assets/exploring-and-analyzing-network-data-with-python/quakers_nodelist.csv\n",
    "    and https://programminghistorian.org/assets/exploring-and-analyzing-network-data-with-python/quakers_edgelist.csv\n",
    "    \n",
    "    - Numerical or analytical results:\n",
    "    The centrality and importance of certain nodes corroborate with the role that this historical persons have \n",
    "    played in the past.\n",
    "    \n",
    "    - Conclusions\n",
    "    The two-way approach (visual and code part) brought similar conclusions that were corroborating. I conclude \n",
    "    that I did the \"basics\" obervations correctly and that I understood the basics principles of a network \n",
    "    analysis.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Introduction\n",
    "- 1/ The \"Religious Society of Friends\", known as the Quakers, was founded in England in the mid-seventeenth century. The Quakers were Protestant Christians who dissented from the official Church of England and who reformed their own religious order in Pennsylvania, US, in mid- to late-seventeenth century.\n",
    "- 2/ The research questions in the studied network are:   \n",
    "     What is the overall network allure?   \n",
    "     Who are the important people, or hubs?   \n",
    "     What are the subgroups and communities?   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Dataset description \n",
    "1/ The links are here : https://programminghistorian.org/assets/exploring-and-analyzing-network-data-with-python/quakers_nodelist.csv  (for the nodes) AND https://programminghistorian.org/assets/exploring-and-analyzing-network-data-with-python/quakers_edgelist.csv (for the links or edges)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Results (and the associated code-process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading files, importing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import networkx as nx\n",
    "from operator import itemgetter\n",
    "import community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# *** program to read CSV files and retrieve the needed data ***\n",
    "\n",
    "# commands for opening and reading our nodelist (early modern Quakers) \n",
    "# and edgelist files (relationships between those Quakers)\n",
    "\n",
    "# Read in the nodelist file\n",
    "with open('quakers_nodelist.csv', 'r') as nodecsv: # Open file (by just \"reading\" it, \"r\")                      \n",
    "    nodereader = csv.reader(nodecsv) # Read csv  \n",
    "    # Retrieve  data (using Python list comprehension and list slicing to remove the header row)\n",
    "    nodes = [n for n in nodereader][1:] # =full nodes list(a node=full info about a given person)                    \n",
    "\n",
    "node_names = [n[0] for n in nodes] # Get a list of only node names (first item in each row)                                      \n",
    "\n",
    "# Read in the edgelist file\n",
    "with open('quakers_edgelist.csv', 'r') as edgecsv: # Open the file (by just \"reading\" it, \"r\")\n",
    "    edgereader = csv.reader(edgecsv) # Read  csv\n",
    "    # Retrieve data (it is a list of tuples)\n",
    "    edges = [tuple(e) for e in edgereader][1:] # (link pairs as tuples = who is connected to who)\n",
    "\n",
    "# Recap : \n",
    "# nodes=  list containing all of the rows from quakers_nodelist.csv, including columns for name, \n",
    "# historical significance, gender, birth year, death year, and SDFB ID\n",
    "# edges = list of tuples (each tuple contains 2 linked people)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of nodes=  119\n"
     ]
    }
   ],
   "source": [
    "print(\"number of nodes= \",len(node_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of links=  174\n"
     ]
    }
   ],
   "source": [
    "print(\"number of links= \",len(edges))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'networkx.classes.graph.Graph'>\n"
     ]
    }
   ],
   "source": [
    "# So far we had 2 Python lists: a list of nodes (node_names) and a list of tuples-links (edges)\n",
    "\n",
    "# *** In NetworkX, 2 lists put together into a single network object = Graph = data organized as a network ***\n",
    "# A Graph understands how nodes and edges are related\n",
    "\n",
    "# Graph object initialization \n",
    "G = nx.Graph() # creates a new Graph object, G,\n",
    "print(type(G))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# addings nodes list and edges list\n",
    "G.add_nodes_from(node_names) # Add nodes to the Graph\n",
    "G.add_edges_from(edges) # Add edges to the Graph  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: \n",
      "Type: Graph\n",
      "Number of nodes: 119\n",
      "Number of edges: 174\n",
      "Average degree:   2.9244\n"
     ]
    }
   ],
   "source": [
    "print(nx.info(G))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attributes adding\n",
    "### (how to create a Graph object and add attributes to it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For NetworkX, a Graph object is one big thing (the network) \n",
    "# made up of two kinds of smaller things (nodes and edges)\n",
    "\n",
    "# NetworkX allows to add attributes to both nodes and edges (providing more information about each of them)\n",
    "\n",
    "\n",
    "\n",
    "# We want to iterate through each of the 2 lists in order to add these informations on the graph.\n",
    "# To do this we first need to create dictionnaries:  node names are the keys and the attributes are the values\n",
    "\n",
    "# create 5 empty dictionaries (with curly braces).\n",
    "# => Create an empty dictionary for each attribute\n",
    "hist_sig_dict = {}\n",
    "gender_dict = {}\n",
    "birth_dict = {}\n",
    "death_dict = {}\n",
    "id_dict = {} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop through nodes list and add  appropriate items to each dictionary\n",
    "\n",
    "\n",
    "# The person’s historical significance will be index 1, their gender will be index 2, and so on\n",
    "for node in nodes: # Loop through the list, one row at a time \n",
    "    hist_sig_dict[node[0]] = node[1] # dictionary[key] = value\n",
    "    gender_dict[node[0]] = node[2]\n",
    "    birth_dict[node[0]] = node[3]\n",
    "    death_dict[node[0]] = node[4]\n",
    "    id_dict[node[0]] = node[5]\n",
    "\n",
    "# check on dictionnay:\n",
    "#print(hist_sig_dict)\n",
    "#print(gender_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we have a set of dictionaries that we can use to add attributes to nodes in our Graph object\n",
    "\n",
    "# => Add each dictionary as a node attribute to the Graph object.\n",
    "# The set_node_attributes function takes three variables: the Graph to which we’re adding the attribute (G), \n",
    "# the dictionary of id-attribute pairs (5 previous lists), and the name of the new attribute (of the 5 lists).\n",
    "nx.set_node_attributes(G, hist_sig_dict, 'historical_significance')\n",
    "nx.set_node_attributes(G, gender_dict, 'gender')\n",
    "nx.set_node_attributes(G, birth_dict, 'birth_year')\n",
    "nx.set_node_attributes(G, death_dict, 'death_year')\n",
    "nx.set_node_attributes(G, id_dict, 'sdfb_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "unexpected EOF while parsing (<ipython-input-26-52f1d62e71d1>, line 12)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-26-52f1d62e71d1>\"\u001b[0;36m, line \u001b[0;32m12\u001b[0m\n\u001b[0;31m    # \"G.nodes[n]['key'] = gives value of this 'key'\u001b[0m\n\u001b[0m                                                    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m unexpected EOF while parsing\n"
     ]
    }
   ],
   "source": [
    "# Now all the nodes have their attributes, accessible any time\n",
    "\n",
    "\n",
    "# For example, we can print out all the birth years of our nodes by looping \n",
    "# through them and accessing the \"birth_year\" attribute, like this:\n",
    "# => Loop through each node, to access and print all the \"birth_year\" attributes\n",
    "for n in G.nodes(): # Loop through every node, in our data \"n\" will be the name of the person\n",
    "    #print(n, G.nodes[n]['birth_year']) # Access every node by its name, and then by the attribute \"birth_year\"\n",
    "\n",
    "# Explanation : \"G.nodes\" = list of all names. \"n\" = names iteration (one after the other one) = person's name\n",
    "# \"G.nodes[n]\" = dictionnary (containing 5 values) attributed to one name\n",
    "# \"G.nodes[n]['key'] = gives value of this 'key'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have an undirected networks (no \"directions\" for the edges are provided = undirected links).\n",
    "# we do not know in which sense the nodges are interacting, we just know they do.\n",
    "\n",
    "# By using the symmetric (undirected relationships in this case) we find sub-communities \n",
    "# and the people who are important to those communities inside the provided network\n",
    "\n",
    "# A network has a topology = connective shape, that could be Centralized OR Decentralized; \n",
    "# Dense OR Sparse; Cyclical OR Linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network density: 0.02478279447372169\n"
     ]
    }
   ],
   "source": [
    "# DENSITY\n",
    "# density = nb of connections that exist / nb connections that could theoritically exist. (it's a likelihood)\n",
    "# calculate network density by \"running nx.density(G)\" \n",
    "# (the best way to do this is to store the metric in a variable and print it). \n",
    "density = nx.density(G)\n",
    "print(\"Network density:\", density) # = 0.0247... so not a lot compared to what is possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shortest path between Fell and Whitehead: ['Margaret Fell', 'George Fox', 'George Whitehead']\n",
      "Length of that path: 2\n"
     ]
    }
   ],
   "source": [
    "# SHORTEST PATH\n",
    "# shorest path = minimum number of people (nodes) that exist between 2 other people.\n",
    "# \"shortest_path\" will be a list of the nodes that includes the “source” (Fell) = start node, to\n",
    "# the “target” (Whitehead) = end node, and all the nodes between them.\n",
    "# (note : could take time to calculate: Python first finds all possible paths and then picks the shortest one)\n",
    "fell_whitehead_path = nx.shortest_path(G, source=\"Margaret Fell\", target=\"George Whitehead\")\n",
    "print(\"Shortest path between Fell and Whitehead:\", fell_whitehead_path)\n",
    "# remarque: there is only \"Fox\" between them. Fox is a hub (=node with many connections)\n",
    "# we can suppose that several shortest paths run through him as a mediator. \n",
    "# What importance the Fox Quaker founder had in this social network?\n",
    "print(\"Length of that path:\", len(fell_whitehead_path)-1) # says that there are 2 \"steps\" between both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "Network diameter of largest component: 8\n"
     ]
    }
   ],
   "source": [
    "# DIAMETER (FOR CONNECTED NETWORK)\n",
    "# network's diameter = largest distance between any nodes pair (longuest of the calculated path lengths)\n",
    "# \"nx.diameter(G)\" = this command on the Quaker graph will yield an error: “not connected”  (not one component)\n",
    "# because graph has more than one component (so there is no bridge-links and there are many different components).\n",
    "\n",
    "# If Graph has more than one component, this will return False (True otherwise):\n",
    "print(nx.is_connected(G))\n",
    "\n",
    "# \"nx.connected_components\"= gets components list (of all components inside network)\n",
    "components = nx.connected_components(G)\n",
    "largest_component = max(components, key=len) # \"max()\"= find the largest component in the list based on the length\n",
    "\n",
    "# Create a 'subgraph' ('subnetwork') of just the largest component (so only the biggest component is keept)\n",
    "subgraph = G.subgraph(largest_component)\n",
    "diameter = nx.diameter(subgraph) # calculate diameter of the subgraph, just like we did with network density\n",
    "print(\"Network diameter of largest component:\", diameter) \n",
    "\n",
    "# note : we took the largest component, we can assume there is no larger diameter for the other components\n",
    "# => There is a path length of 8 between the 2 farthest-apart nodes in the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Triadic closure: 0.16937799043062202\n"
     ]
    }
   ],
   "source": [
    "# TRIADIC CLOSURE (measures: clustering coefficient OR transitivity)\n",
    "# Triadic closure: if two people know the same person, they are likely to know each other. \n",
    "# If Fox knows both Fell and Whitehead, then Fell and Whitehead may very well know each other, completing \n",
    "# a triangle (in the visualization of three edges connecting Fox, Fell, and Whitehead). \n",
    "# => The number of these enclosed triangles in the network can be used to find clusters and communities \n",
    "# of individuals that all know each other\n",
    "\n",
    "# TRANSITIVITY (a triadic closure measure) = ratio of all triangles observed / all possible triangles in theory\n",
    "# = expresses how interconnected a graph is (in terms of a ratio of actual over possible connections).\n",
    "# (it's a likelihood). Like density, transitivity is scaled from 0 to 1\n",
    "triadic_closure = nx.transitivity(G)\n",
    "print(\"Triadic closure:\", triadic_closure)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Centrality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# “Which nodes are the most important?”. In network analysis, measures of the importance of nodes are referred \n",
    "# to as centrality measures. There are many different ways of calculating centrality.\n",
    "# 3 of the most common centrality measures: degree, betweenness centrality, and eigenvector centrality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All of the centrality commands in this section produce dictionaries \n",
    "# in which the keys are nodes and the values are centrality measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22\n"
     ]
    }
   ],
   "source": [
    "# DEGREE (CENTRALITY)\n",
    "# A node’s degree is the sum of its edges (its number of links)\n",
    "# nodes with the highest degree in a social network = people who know the most people = hubs\n",
    "\n",
    "# calculating degree and adding it as an attribute to the network\n",
    "# \"G.degree()\" method calculates degree for all the nodes in put it in a tuple => \"dict\" tranfsorms it in dictio\n",
    "# \"degree_dict\" contains all nodes names (key) and their degree(value) in a dictionnary (for each different node)\n",
    "degree_dict = dict(G.degree(G.nodes())) # runs \"G.degree()\" method on the full G.nodes() list (node names)\n",
    "# \"set_node_attributes(G, values, name=None)\" with G = NetworkX Graph ,values = dictionnary or value, and 'name'\n",
    "# \"set_node_attributes(G, values, name=None)\" = Sets node attributes from a given value or dictionary of values.\n",
    "nx.set_node_attributes(G, degree_dict, 'degree')\n",
    "\n",
    "# check it out (to debug):\n",
    "#print(G.nodes())\n",
    "#print(G.degree(G.nodes())) \n",
    "#print(degree_dict)\n",
    "\n",
    "# and we can see and check a degree for a specific node:\n",
    "print(G.nodes['George Fox']['degree'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 20 nodes by degree:\n",
      "('George Fox', 22)\n",
      "('William Penn', 18)\n",
      "('James Nayler', 16)\n",
      "('George Whitehead', 13)\n",
      "('Margaret Fell', 13)\n",
      "('Benjamin Furly', 10)\n",
      "('Edward Burrough', 9)\n",
      "('George Keith', 8)\n",
      "('Thomas Ellwood', 8)\n",
      "('Francis Howgill', 7)\n",
      "('John Perrot', 7)\n",
      "('John Audland', 6)\n",
      "('Richard Farnworth', 6)\n",
      "('Alexander Parker', 6)\n",
      "('John Story', 6)\n",
      "('John Stubbs', 5)\n",
      "('Thomas Curtis', 5)\n",
      "('John Wilkinson', 5)\n",
      "('William Caton', 5)\n",
      "('Anthony Pearson', 5)\n"
     ]
    }
   ],
   "source": [
    "# \"sorted()\" = built-in function = sorts a dictionary by its keys or values and find the top twenty nodes \n",
    "# ranked by degree. Need to use \"itemgetter\", imported back at the beginning of the tutorial\n",
    "sorted_degree = sorted(degree_dict.items(), key=itemgetter(1), reverse=True)\n",
    "# dictionary we want to sort=  degree_dict.items(). \n",
    "# The second, what to sort by: item “1” is the second item in the pair = value of the dictionary. \n",
    "# it sorts from small to big, so we reverse to have big->small\n",
    "\n",
    "# check it out here:\n",
    "#print(degree_dict)\n",
    "#print(degree_dict.items())\n",
    "#print(sorted_degree)\n",
    "\n",
    "print(\"Top 20 nodes by degree:\")\n",
    "for d in sorted_degree[:20]:\n",
    "    print(d) # yes indeed, we see that the hubs are the know historically important persons\n",
    "# but further analysis could be important as well to go bit deeper..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 20 nodes by eigenvector centrality:\n",
      "('George Fox', 0.4491750710859924)\n",
      "('James Nayler', 0.3352974100447867)\n",
      "('William Penn', 0.2703220115399868)\n",
      "('Margaret Fell', 0.253170949905681)\n",
      "('George Whitehead', 0.2497455334914196)\n",
      "('Edward Burrough', 0.23147427604862297)\n",
      "('Francis Howgill', 0.1909539378268105)\n",
      "('Benjamin Furly', 0.1878520634691651)\n",
      "('John Perrot', 0.1849692807795611)\n",
      "('George Keith', 0.18384690867915351)\n",
      "('Thomas Ellwood', 0.17608142535843857)\n",
      "('Richard Farnworth', 0.15368535029296415)\n",
      "('John Crook', 0.1327158126880779)\n",
      "('Rebecca Travers', 0.1184804064465093)\n",
      "('Alexander Parker', 0.11587808682088324)\n",
      "('Anthony Pearson', 0.11120476725256784)\n",
      "('William Dewsbury', 0.11057869321157121)\n",
      "('John Stubbs', 0.10693500692141825)\n",
      "('John Audland', 0.0983088971933375)\n",
      "('Thomas Salthouse', 0.09548628544138771)\n"
     ]
    }
   ],
   "source": [
    "# EIGENVECTOR:\n",
    "# Eigenvector centrality cares if you are a hub, but it also cares how many hubs you are connected to. \n",
    "# It’s calculated as a value from 0 to 1: the closer to one, the greater the centrality. \n",
    "# Eigenvector centrality is useful for understanding which nodes can get information to many other nodes quickly\n",
    "eigenvector_dict = nx.eigenvector_centrality(G) # Run eigenvector centrality\n",
    "# \"set_node_attributes(G, values, name=None)\" with G = NetworkX Graph ,values = dictionnary or value, and 'name'\n",
    "nx.set_node_attributes(G, eigenvector_dict, 'eigenvector')\n",
    "\n",
    "sorted_eigenvector = sorted(eigenvector_dict.items(), key=itemgetter(1), reverse=True)\n",
    "\n",
    "print(\"Top 20 nodes by eigenvector centrality:\")\n",
    "for e in sorted_eigenvector[:20]:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 20 nodes by betweenness centrality:\n",
      "('William Penn', 0.23999456006192205)\n",
      "('George Fox', 0.23683257726065216)\n",
      "('George Whitehead', 0.12632024847366005)\n",
      "('Margaret Fell', 0.12106792237170329)\n",
      "('James Nayler', 0.10446026280446098)\n",
      "('Benjamin Furly', 0.06419626175167242)\n",
      "('Thomas Ellwood', 0.046190623885104545)\n",
      "('George Keith', 0.045006564009171565)\n",
      "('John Audland', 0.04164936340077581)\n",
      "('Alexander Parker', 0.03893676140525336)\n",
      "('John Story', 0.028990098622866983)\n",
      "('John Burnyeat', 0.028974117533439564)\n",
      "('John Perrot', 0.02829566854990583)\n",
      "('James Logan', 0.026944806605823553)\n",
      "('Richard Claridge', 0.026944806605823553)\n",
      "('Robert Barclay', 0.026944806605823553)\n",
      "('Elizabeth Leavens', 0.026944806605823553)\n",
      "('Thomas Curtis', 0.026729751729751724)\n",
      "('John Stubbs', 0.024316593960227152)\n",
      "('Mary Penington', 0.02420824624214454)\n"
     ]
    }
   ],
   "source": [
    "# Betweenness centrality (=> to find BROKERS)\n",
    "# Betweenness centrality looks at all the shortest paths that pass through a particular node.\n",
    "#  it must first calculate every possible shortest path in the network (can take some time).\n",
    "# => Quick way of giving a sense of which nodes are important not because they have lots of connections themselves\n",
    "# but because they stand between groups, giving the network connectivity and cohesion = BROKER\n",
    "betweenness_dict = nx.betweenness_centrality(G) # Run betweenness centrality\n",
    "# \"set_node_attributes(G, values, name=None)\" with G = NetworkX Graph ,values = dictionnary or value, and 'name'\n",
    "nx.set_node_attributes(G, betweenness_dict, 'betweenness')\n",
    "\n",
    "sorted_betweenness = sorted(betweenness_dict.items(), key=itemgetter(1), reverse=True)\n",
    "\n",
    "print(\"Top 20 nodes by betweenness centrality:\")\n",
    "for b in sorted_betweenness[:20]:\n",
    "    print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: William Penn | Betweenness Centrality: 0.23999456006192205 | Degree: 18\n",
      "Name: George Fox | Betweenness Centrality: 0.23683257726065216 | Degree: 22\n",
      "Name: George Whitehead | Betweenness Centrality: 0.12632024847366005 | Degree: 13\n",
      "Name: Margaret Fell | Betweenness Centrality: 0.12106792237170329 | Degree: 13\n",
      "Name: James Nayler | Betweenness Centrality: 0.10446026280446098 | Degree: 16\n",
      "Name: Benjamin Furly | Betweenness Centrality: 0.06419626175167242 | Degree: 10\n",
      "Name: Thomas Ellwood | Betweenness Centrality: 0.046190623885104545 | Degree: 8\n",
      "Name: George Keith | Betweenness Centrality: 0.045006564009171565 | Degree: 8\n",
      "Name: John Audland | Betweenness Centrality: 0.04164936340077581 | Degree: 6\n",
      "Name: Alexander Parker | Betweenness Centrality: 0.03893676140525336 | Degree: 6\n",
      "Name: John Story | Betweenness Centrality: 0.028990098622866983 | Degree: 6\n",
      "Name: John Burnyeat | Betweenness Centrality: 0.028974117533439564 | Degree: 4\n",
      "Name: John Perrot | Betweenness Centrality: 0.02829566854990583 | Degree: 7\n",
      "Name: James Logan | Betweenness Centrality: 0.026944806605823553 | Degree: 4\n",
      "Name: Richard Claridge | Betweenness Centrality: 0.026944806605823553 | Degree: 2\n",
      "Name: Robert Barclay | Betweenness Centrality: 0.026944806605823553 | Degree: 3\n",
      "Name: Elizabeth Leavens | Betweenness Centrality: 0.026944806605823553 | Degree: 2\n",
      "Name: Thomas Curtis | Betweenness Centrality: 0.026729751729751724 | Degree: 5\n",
      "Name: John Stubbs | Betweenness Centrality: 0.024316593960227152 | Degree: 5\n",
      "Name: Mary Penington | Betweenness Centrality: 0.02420824624214454 | Degree: 4\n"
     ]
    }
   ],
   "source": [
    "#First get the top 20 nodes by betweenness as a list\n",
    "top_betweenness = sorted_betweenness[:20]\n",
    "\n",
    "#Then find and print their degree\n",
    "for tb in top_betweenness: # Loop through the top_betweenness list above\n",
    "    degree = degree_dict[tb[0]] # Use degree_dict to access a node's degree (see footnote 2)\n",
    "    print(\"Name:\", tb[0], \"| Betweenness Centrality:\", tb[1], \"| Degree:\", degree)\n",
    "# explanation: tb[0] = key (name) and tb[1] = value (betweenness-centrality) of top_betweenness list \n",
    "\n",
    "# Note: Penn has lower degree than Quaker founder George Fox, but higher betweenness centrality. \n",
    "# simply knowing more people isn’t everything, he was also between different people "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Community detection with modularity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There are many ways of calculating communities, cliques, and clusters in your network, \n",
    "# but the most popular method currently is modularity. \n",
    "# Modularity = measure of relative density in a network. Modularity gives an overall score of how fractious a \n",
    "# network is; That score can be used to partition the network and return the individual communities.\n",
    "# A community = module = modularity class = has high density with nodes within its module but \n",
    "# low density with those outside. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0: ['Joseph Wyeth', 'Thomas Curtis', 'William Rogers', 'John Penington', 'Mary Penington', 'Thomas Ellwood', 'William Simpson']\n",
      "Class 2: ['James Logan', 'Richard Claridge', 'William Bradford', 'Isabel Yeamans', 'Samuel Clarridge', 'James Claypoole', 'Joseph Besse', 'Jane Sowle', 'Tace Sowle', 'Peter Collinson', 'Isaac Norris', 'Anthony Sharp', 'John Bartram', 'Edward Haistwell', 'John ap John', 'John Burnyeat', 'William Edmundson', 'William Penn', 'David Lloyd', 'Thomas Story', 'Samuel Bownas']\n",
      "Class 3: ['Dorcas Erbery', 'Gervase Benson', 'James Nayler', 'William Crouch', 'Robert Rich', 'Richard Farnworth', 'Thomas Aldam', 'Francis Howgill', 'Richard Hubberthorne', 'William Tomlinson', 'Anthony Pearson', 'Martha Simmonds', 'Hannah Stranger']\n",
      "Class 4: ['William Mucklow', 'William Dewsbury', 'George Fox', 'John Crook', 'Ellis Hookes', 'Elizabeth Hooten', 'William Coddington', 'Leonard Fell', 'Mary Fisher', 'Mary Prince', 'Edward Burrough', 'John Perrot']\n",
      "Class 5: ['Thomas Salthouse', 'George Fox the younger', 'Thomas Lower', 'Thomas Holme', 'William Mead', 'Thomas Lawson', 'Margaret Fell', 'Elizabeth Leavens', 'Alexander Parker', 'Sir Charles Wager', 'William Gibson', 'Lewis Morris']\n",
      "Class 6: ['John Audland', 'Anne Camm', 'John Camm', 'Thomas Camm', 'John Wilkinson', 'Solomon Eccles', 'Edward Pyott', 'Charles Marshall', 'John Story']\n",
      "Class 8: ['John Stubbs', 'Stephen Crisp', 'John Swinton', 'Benjamin Furly', 'Robert Barclay', 'David Barclay of Ury', 'George Keith', 'James Parnel', 'Franciscus Mercurius van Helmont', 'William Caton', 'William Ames', 'Anne Conway Viscountess Conway and Killultagh', 'Samuel Fisher']\n",
      "Class 12: ['Henry Pickworth', 'Gilbert Latey', 'George Whitehead', 'John Whitehead', 'Silvanus Bevan', 'Alice Curwen', 'Francis Bugg', 'Rebecca Travers', 'Daniel Quare']\n",
      "Class 13: ['John Whiting', 'Christopher Taylor', 'Thomas Taylor']\n"
     ]
    }
   ],
   "source": [
    "# COMMUNITY DETECTION AND PARTITIONING (we use the \"community\" library, imported above)\n",
    "# \"best_partition()\" tries to determine the number of communities appropriate for the graph\n",
    "#  and assigns each node a number (starting at 0), corresponding to the community it is a member of\n",
    "communities = community.best_partition(G) # create a dictionary just like the ones created by centrality functions\n",
    "\n",
    "# adds these values to the network\n",
    "# \"set_node_attributes(G, values, name=None)\" with G = NetworkX Graph ,values = dictionnary or value, and 'name'\n",
    "nx.set_node_attributes(G, communities, 'modularity')\n",
    "\n",
    "# We put everything in CLASSES and we select COMMUNITIES > 2 MEMBERS (NODES)\n",
    "modularity = {} # Create a new empty dictionary\n",
    "for k,v in communities.items(): # Loop through the community dictionary (v = value = nomber and k = key = name)\n",
    "    if v not in modularity:\n",
    "        modularity[v] = [k] # Add a new key for a modularity class (if it was not inside)\n",
    "    else:\n",
    "        modularity[v].append(k) # Append a name to the list for a modularity class (the code has already seen)\n",
    "\n",
    "for k,v in modularity.items(): # Loop through the new dictionary \"modularity\"\n",
    "    if len(v) > 2: # Filter out modularity classes with 2 or fewer nodes \n",
    "        print('Class '+str(k)+':', v) # Print out the classes and their members"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Conclussions\n",
    "   - 8 classes with more than 2 members\n",
    "   - few very important node within this networks, but with many \"smaller\" nodes that have an important role to \n",
    "     play (like those who have an important betweenness centrality)\n",
    "   - Please see the slides in the class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ** Open questions\n",
    "Can we develop a NLP tool that would analyse scrapped informations from the internet to see if the obervations match with the results of our study?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}