Python_Course_MeInBio_PART3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"IMAGES_PYTHON_COURSE_2018/LOGO_MEINBIO_TRANSPARENT.png\" width=\"200\" height=\"200\" />\n",
    "# <center>Python course 2018\n",
    "### <center>MeInBio Training Group\n",
    "#### <center> by Florian Heyl & Francesco Ferrari \n",
    "Welcome to this introduction to Python. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART 3 : READ AND WRITE FILES\n",
    "\n",
    "Reading and writing files is a daily bread for a data scientist. \n",
    "\n",
    "Examples of text files which you\n",
    "might have encountered or will encounter in the future include:\n",
    "\n",
    "• FASTA/FASTQ files of DNA or protein sequences <br>\n",
    "• files containing output from command-line programs (e.g. BLAST) <br>\n",
    "• HTML files <br>\n",
    "• word processing documents <br>\n",
    "• table data in comma (csv) ot tab (tsv) separated format <br>\n",
    "• and of course, Python codet. <br>\n",
    "\n",
    "We provided for you some test data in the folder \"test_data\" to play with. \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: Reading Files\n",
    "Lets start with a simple txt file that you might encounter when you want to extend intervals in a genome. First of all you need to know how far you can extend it. So let's try and get the information that we need.\n",
    "\n",
    "The information is written in the file **\"test_data/hg19_chr_sizes.txt\"**. So let's retrieve the information of that file. For that, in every language, we need to open the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_file = open(\"test_data/hg19_chr_sizes.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is now buffered inside the variable **my_file** with is a new object holding the data. The object **my_file** has to be handled a bit differently than the previous variables we played with so far. Every operation has to be done with functions that can deal with this type of object. In order to get the data in python we do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr4\t191154276\n",
      "\n"
     ]
    }
   ],
   "source": [
    "first_line = my_file.readline()\n",
    "print(first_line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cool thing about jupiter is the interactive nature. <br/>\n",
    "\n",
    "<img src=\"IMAGES_PYTHON_COURSE_2018/question-mark.png\" width=\"24\" height=\"24\" style=\"float: left;\"/> Try to execute the previous command a couple of times. What do you witness?\n",
    "\n",
    "### TASK:\n",
    "\n",
    "Let's do some execise on the data you just safed in **first_line**. Normally you have to process data further. Your task is now to safe the chromosome id in one variable called **c** and the size in another variable **s**. But wait, first check what this **first_line** is actually. In python you can check the type of variable with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "print(type(first_line))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"IMAGES_PYTHON_COURSE_2018/question-mark.png\" width=\"24\" height=\"24\" style=\"float: left;\"/> What does the type name stand for?\n",
    "\n",
    "As we figured out the type of **first_line**, we can simply apply operations we already learned from the previous exercises. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Reading Files and for loops\n",
    "\n",
    "You already experienced some way to iterate thing with **for loops**. One question that might came up in your mind as you solved the previous task was: \"What about the whole file?\" So far we only parsed one line of the whole. So, let's dig into a more complex task and print everything.\n",
    "\n",
    "For that you need to understand that **for loops** in python can have special \"abilities\" based on the object you deal with. For our text-file object, for example, we can define a **for loop** that will read every line out of our file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr5\t180915260\n",
      "\n",
      "chr6\t171115067\n",
      "\n",
      "chr7\t159138663\n",
      "\n",
      "chrX\t155270560\n",
      "\n",
      "chr8\t146364022\n",
      "\n",
      "chr9\t141213431\n",
      "\n",
      "chr10\t135534747\n",
      "\n",
      "chr11\t135006516\n",
      "\n",
      "chr12\t133851895\n",
      "\n",
      "chr13\t115169878\n",
      "\n",
      "chr14\t107349540\n",
      "\n",
      "chr15\t102531392\n",
      "\n",
      "chr16\t90354753\n",
      "\n",
      "chr17\t81195210\n",
      "\n",
      "chr18\t78077248\n",
      "\n",
      "chr20\t63025520\n",
      "\n",
      "chrY\t59373566\n",
      "\n",
      "chr19\t59128983\n",
      "\n",
      "chr22\t51304566\n",
      "\n",
      "chr21\t48129895\n",
      "\n",
      "chr6_ssto_hap7\t4928567\n",
      "\n",
      "chr6_mcf_hap5\t4833398\n",
      "\n",
      "chr6_cox_hap2\t4795371\n",
      "\n",
      "chr6_mann_hap4\t4683263\n",
      "\n",
      "chr6_apd_hap1\t4622290\n",
      "\n",
      "chr6_qbl_hap6\t4611984\n",
      "\n",
      "chr6_dbb_hap3\t4610396\n",
      "\n",
      "chr17_ctg5_hap1\t1680828\n",
      "\n",
      "chr4_ctg9_hap1\t590426\n",
      "\n",
      "chr1_gl000192_random\t547496\n",
      "\n",
      "chrUn_gl000225\t211173\n",
      "\n",
      "chr4_gl000194_random\t191469\n",
      "\n",
      "chr4_gl000193_random\t189789\n",
      "\n",
      "chr9_gl000200_random\t187035\n",
      "\n",
      "chrUn_gl000222\t186861\n",
      "\n",
      "chrUn_gl000212\t186858\n",
      "\n",
      "chr7_gl000195_random\t182896\n",
      "\n",
      "chrUn_gl000223\t180455\n",
      "\n",
      "chrUn_gl000224\t179693\n",
      "\n",
      "chrUn_gl000219\t179198\n",
      "\n",
      "chr17_gl000205_random\t174588\n",
      "\n",
      "chrUn_gl000215\t172545\n",
      "\n",
      "chrUn_gl000216\t172294\n",
      "\n",
      "chrUn_gl000217\t172149\n",
      "\n",
      "chr9_gl000199_random\t169874\n",
      "\n",
      "chrUn_gl000211\t166566\n",
      "\n",
      "chrUn_gl000213\t164239\n",
      "\n",
      "chrUn_gl000220\t161802\n",
      "\n",
      "chrUn_gl000218\t161147\n",
      "\n",
      "chr19_gl000209_random\t159169\n",
      "\n",
      "chrUn_gl000221\t155397\n",
      "\n",
      "chrUn_gl000214\t137718\n",
      "\n",
      "chrUn_gl000228\t129120\n",
      "\n",
      "chrUn_gl000227\t128374\n",
      "\n",
      "chr1_gl000191_random\t106433\n",
      "\n",
      "chr19_gl000208_random\t92689\n",
      "\n",
      "chr9_gl000198_random\t90085\n",
      "\n",
      "chr17_gl000204_random\t81310\n",
      "\n",
      "chrUn_gl000233\t45941\n",
      "\n",
      "chrUn_gl000237\t45867\n",
      "\n",
      "chrUn_gl000230\t43691\n",
      "\n",
      "chrUn_gl000242\t43523\n",
      "\n",
      "chrUn_gl000243\t43341\n",
      "\n",
      "chrUn_gl000241\t42152\n",
      "\n",
      "chrUn_gl000236\t41934\n",
      "\n",
      "chrUn_gl000240\t41933\n",
      "\n",
      "chr17_gl000206_random\t41001\n",
      "\n",
      "chrUn_gl000232\t40652\n",
      "\n",
      "chrUn_gl000234\t40531\n",
      "\n",
      "chr11_gl000202_random\t40103\n",
      "\n",
      "chrUn_gl000238\t39939\n",
      "\n",
      "chrUn_gl000244\t39929\n",
      "\n",
      "chrUn_gl000248\t39786\n",
      "\n",
      "chr8_gl000196_random\t38914\n",
      "\n",
      "chrUn_gl000249\t38502\n",
      "\n",
      "chrUn_gl000246\t38154\n",
      "\n",
      "chr17_gl000203_random\t37498\n",
      "\n",
      "chr8_gl000197_random\t37175\n",
      "\n",
      "chrUn_gl000245\t36651\n",
      "\n",
      "chrUn_gl000247\t36422\n",
      "\n",
      "chr9_gl000201_random\t36148\n",
      "\n",
      "chrUn_gl000235\t34474\n",
      "\n",
      "chrUn_gl000239\t33824\n",
      "\n",
      "chr21_gl000210_random\t27682\n",
      "\n",
      "chrUn_gl000231\t27386\n",
      "\n",
      "chrUn_gl000229\t19913\n",
      "\n",
      "chrM\t16571\n",
      "\n",
      "chrUn_gl000226\t15008\n",
      "\n",
      "chr18_gl000207_random\t4262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for line in my_file:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"IMAGES_PYTHON_COURSE_2018/question-mark.png\" width=\"24\" height=\"24\" style=\"float: left;\"/> Why don't we see everything from the file?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might figured it out already. As mentioned previously, the **my_file** variable (object) is a buffer holding our data. When we read from the buffer we also remove the information. So we need to open the file again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr1\t249250621\n",
      "\n",
      "chr2\t243199373\n",
      "\n",
      "chr3\t198022430\n",
      "\n",
      "chr4\t191154276\n",
      "\n",
      "chr5\t180915260\n",
      "\n",
      "chr6\t171115067\n",
      "\n",
      "chr7\t159138663\n",
      "\n",
      "chrX\t155270560\n",
      "\n",
      "chr8\t146364022\n",
      "\n",
      "chr9\t141213431\n",
      "\n",
      "chr10\t135534747\n",
      "\n",
      "chr11\t135006516\n",
      "\n",
      "chr12\t133851895\n",
      "\n",
      "chr13\t115169878\n",
      "\n",
      "chr14\t107349540\n",
      "\n",
      "chr15\t102531392\n",
      "\n",
      "chr16\t90354753\n",
      "\n",
      "chr17\t81195210\n",
      "\n",
      "chr18\t78077248\n",
      "\n",
      "chr20\t63025520\n",
      "\n",
      "chrY\t59373566\n",
      "\n",
      "chr19\t59128983\n",
      "\n",
      "chr22\t51304566\n",
      "\n",
      "chr21\t48129895\n",
      "\n",
      "chr6_ssto_hap7\t4928567\n",
      "\n",
      "chr6_mcf_hap5\t4833398\n",
      "\n",
      "chr6_cox_hap2\t4795371\n",
      "\n",
      "chr6_mann_hap4\t4683263\n",
      "\n",
      "chr6_apd_hap1\t4622290\n",
      "\n",
      "chr6_qbl_hap6\t4611984\n",
      "\n",
      "chr6_dbb_hap3\t4610396\n",
      "\n",
      "chr17_ctg5_hap1\t1680828\n",
      "\n",
      "chr4_ctg9_hap1\t590426\n",
      "\n",
      "chr1_gl000192_random\t547496\n",
      "\n",
      "chrUn_gl000225\t211173\n",
      "\n",
      "chr4_gl000194_random\t191469\n",
      "\n",
      "chr4_gl000193_random\t189789\n",
      "\n",
      "chr9_gl000200_random\t187035\n",
      "\n",
      "chrUn_gl000222\t186861\n",
      "\n",
      "chrUn_gl000212\t186858\n",
      "\n",
      "chr7_gl000195_random\t182896\n",
      "\n",
      "chrUn_gl000223\t180455\n",
      "\n",
      "chrUn_gl000224\t179693\n",
      "\n",
      "chrUn_gl000219\t179198\n",
      "\n",
      "chr17_gl000205_random\t174588\n",
      "\n",
      "chrUn_gl000215\t172545\n",
      "\n",
      "chrUn_gl000216\t172294\n",
      "\n",
      "chrUn_gl000217\t172149\n",
      "\n",
      "chr9_gl000199_random\t169874\n",
      "\n",
      "chrUn_gl000211\t166566\n",
      "\n",
      "chrUn_gl000213\t164239\n",
      "\n",
      "chrUn_gl000220\t161802\n",
      "\n",
      "chrUn_gl000218\t161147\n",
      "\n",
      "chr19_gl000209_random\t159169\n",
      "\n",
      "chrUn_gl000221\t155397\n",
      "\n",
      "chrUn_gl000214\t137718\n",
      "\n",
      "chrUn_gl000228\t129120\n",
      "\n",
      "chrUn_gl000227\t128374\n",
      "\n",
      "chr1_gl000191_random\t106433\n",
      "\n",
      "chr19_gl000208_random\t92689\n",
      "\n",
      "chr9_gl000198_random\t90085\n",
      "\n",
      "chr17_gl000204_random\t81310\n",
      "\n",
      "chrUn_gl000233\t45941\n",
      "\n",
      "chrUn_gl000237\t45867\n",
      "\n",
      "chrUn_gl000230\t43691\n",
      "\n",
      "chrUn_gl000242\t43523\n",
      "\n",
      "chrUn_gl000243\t43341\n",
      "\n",
      "chrUn_gl000241\t42152\n",
      "\n",
      "chrUn_gl000236\t41934\n",
      "\n",
      "chrUn_gl000240\t41933\n",
      "\n",
      "chr17_gl000206_random\t41001\n",
      "\n",
      "chrUn_gl000232\t40652\n",
      "\n",
      "chrUn_gl000234\t40531\n",
      "\n",
      "chr11_gl000202_random\t40103\n",
      "\n",
      "chrUn_gl000238\t39939\n",
      "\n",
      "chrUn_gl000244\t39929\n",
      "\n",
      "chrUn_gl000248\t39786\n",
      "\n",
      "chr8_gl000196_random\t38914\n",
      "\n",
      "chrUn_gl000249\t38502\n",
      "\n",
      "chrUn_gl000246\t38154\n",
      "\n",
      "chr17_gl000203_random\t37498\n",
      "\n",
      "chr8_gl000197_random\t37175\n",
      "\n",
      "chrUn_gl000245\t36651\n",
      "\n",
      "chrUn_gl000247\t36422\n",
      "\n",
      "chr9_gl000201_random\t36148\n",
      "\n",
      "chrUn_gl000235\t34474\n",
      "\n",
      "chrUn_gl000239\t33824\n",
      "\n",
      "chr21_gl000210_random\t27682\n",
      "\n",
      "chrUn_gl000231\t27386\n",
      "\n",
      "chrUn_gl000229\t19913\n",
      "\n",
      "chrM\t16571\n",
      "\n",
      "chrUn_gl000226\t15008\n",
      "\n",
      "chr18_gl000207_random\t4262\n",
      "\n"
     ]
    }
   ],
   "source": [
    "my_file2 = open(\"test_data/hg19_chr_sizes.txt\")\n",
    "for line in my_file2:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like we want it! The content of the file is now printed in the standard output. \n",
    "\n",
    "### TASK\n",
    "\n",
    "Like we tried in exercise 1, amend the for loop and split every line first printing the chromosome and then the size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: Closing  a File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Closing is something which is quite common for different programming languages. Some languages like python are very kind when it comes to closing. Lets see why closing might be important.\n",
    "\n",
    "### TASK\n",
    "\n",
    "Try to read another line from **my_file2** and print it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Whops what happend?** \n",
    "    Do you remember what we learned about the file object. We have read everything that was loaded in the buffer. The buffer is therefore empty. If you try now and retrieve another line, then the buffer just gives you an empty string. This can raise actually some problems in bigger scripts. Imagine you forgot that you read the file in the beginning of this exercise, then you are confused why you don't get anything back from it in the second attempt.\n",
    "\n",
    "for that reason we have the **close()** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_file2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Closing means, that we delete the content of the buffer permanently. \n",
    "\n",
    "### TASK\n",
    "\n",
    "Try to read another line from **my_file2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you get an error, informing you that the file was closed. This helps you to organise your code. \n",
    "\n",
    "### TASK\n",
    "\n",
    "In the beginning, we had the variable **my_file**. We then used **my_file2** to open the file again.   \n",
    "Take the for loop from exercise 2. Your task is now to open the file **\"test_data/hg19_chr_sizes.txt\"** and print **only** the chromosome names. Then open the file a second time and **only** print the sizes. **Just use one variable, e.g., my_file, to do both for loops!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always remember to close a file. This iy really crucial and even more devasting for other programming languages like C++."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: Writing a File\n",
    "\n",
    "Writing a file is as simple as reading a file. Let's open a file for writing first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_file = open(\"test_data/helloworld.txt\", \"w\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now opened a new text file called **helloworld** in the test_data folder. The argument \"w\" in open() means that we opened a file just to write in it. Per default python opens a file with open() just to read it. Take a look in the folder \"test_data\" you will see that the file **helloworld.txt** is created. Lets fill it with some content. To write to a file we simply use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_file.write(\"Hello World!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look inside of the file **helloworld.txt**. Whops! Did we make a mistake.<br/>\n",
    "\n",
    "<img src=\"IMAGES_PYTHON_COURSE_2018/question-mark.png\" width=\"24\" height=\"24\" style=\"float: left;\"/> There is nothing in it, but why?\n",
    "\n",
    "Like reading, the **new_file** variable is just a buffer holding our newly writiing content inside. For memory and speed efficiency it will write everything to our newly defined text file as soon as we close it. So lets append it further and see what will when we close **new_file**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_file.write(\"><(((°>\")\n",
    "new_file.write(\":)\")\n",
    "new_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"IMAGES_PYTHON_COURSE_2018/question-mark.png\" width=\"24\" height=\"24\" style=\"float: left;\"/> Again, something is not right. I wanted three seperated line but everything got squished in one line. What have I done wrong?\n",
    "\n",
    "### TASK\n",
    "\n",
    "Open the file the **helloworld.txt** again and print the defined output in the lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also append a file by replacing the argument \"w\" with \"a\". \n",
    "\n",
    "### TASK\n",
    "\n",
    "Open the file **helloworld.txt** again and **add** the line \"Am I seriously having fun?\" to it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concluding Exercise\n",
    "\n",
    "With everything you have learned so far, try to solve the following task:\n",
    "\n",
    "Take the data **test_data/hg19_chr_sizes.txt** and create two new text files from it. The first text file **chromosomes.txt** holds the chromosome names and the second **sizes.txt** hold the corresponding chromosome sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concluding Exercise 2\n",
    "\n",
    "Read the file **test_peaks.csv** in test_data. Create a new file **significant_peaks.csv** where you write every peak (peak_id, log2foldchange, p-value) for those peaks that have a log2foldchange **greater or equal to 1** and a p-value **smaller or equal to 0.01**. The columns you need are (2,12,16)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### put your code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}