-
Notifications
You must be signed in to change notification settings - Fork 9
/
03-data-types-subsetting.html
271 lines (271 loc) · 23.2 KB
/
03-data-types-subsetting.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<title>Software Carpentry: R for reproducible scientific analysis</title>
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap.css" />
<link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap-theme.css" />
<link rel="stylesheet" type="text/css" href="css/swc.css" />
<link rel="alternate" type="application/rss+xml" title="Software Carpentry Blog" href="http://software-carpentry.org/feed.xml"/>
<meta charset="UTF-8" />
<!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body class="lesson">
<div class="container card">
<div class="banner">
<a href="http://software-carpentry.org" title="Software Carpentry">
<img alt="Software Carpentry banner" src="img/software-carpentry-banner.png" />
</a>
</div>
<article>
<div class="row">
<div class="col-md-10 col-md-offset-1">
<a href="index.html"><h1 class="title">R for reproducible scientific analysis</h1></a>
<h2 class="subtitle">Vectors & Data frames</h2>
<section class="objectives panel panel-warning">
<div class="panel-heading">
<h2 id="learning-objectives"><span class="glyphicon glyphicon-certificate"></span>Learning objectives</h2>
</div>
<div class="panel-body">
<ul>
<li>Understand <code>data.frame</code> as common R object and familiar research data table.
<ul>
<li>Columns = variables, rows = observations</li>
<li>Columns = vectors</li>
</ul></li>
<li>Be able to examine the structure and content of a data frame.</li>
<li>Understand vectors as fundamental R unit</li>
<li>Understand and be able to use vectorized operations</li>
<li>Be able to subset vectors with index and logical notation</li>
<li>Be able to import .RDA and .CSV files</li>
</ul>
</div>
</section>
<h3 id="loading-data">Loading data</h3>
<p>R has a variety functions to load specific kinds of data, e.g. <code>read.csv</code> to read a comma-separated value table. Two R-specific data-types are <code>.RDA</code> and <code>.RDS</code>. <code>.RDA</code> files, like the one you just saved, are created with the <code>save</code> function and accessed with the <code>load</code> function. Load needs the location of the saved file, provided as character string file-path, starting with the working directory. We won’t get into working directories now, except to say that RStudio’s projects makes this easier, by making the location of your project (where the <code>.Rproj</code> file is located) the default working directory. It is displayed at the top of your console pane in RStudio. File-paths should be provided relative to that location. So, to load the file we just saved:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">load</span>(<span class="st">'data/continents.RDA'</span>)</code></pre></div>
<h3 id="intro-to-data.frames">Intro to data.frames</h3>
<p>Check your “Environment” tab – a new object called <code>continents</code> has appeared. If your environment is in “list” mode, switch to “grid” as it’s more informative. We can see that <code>continents</code> has Type data.frame and contains 6 observations of 4 variables. Click on the spreadsheet icon at far-right to view it.</p>
<p>This is a <code>data.frame</code> – the type of object most of us work with most of the time in R. In a <code>data.frame</code> each column represents a variable, and each row is an observation. This is the basic data organizational scheme – one column per variable, one row per observation. You might recognize this form from from a statistics class or your own data analysis.</p>
<h4 id="inspecting-a-data.frame">Inspecting a data.frame</h4>
<p>Rather than pulling up the spreadsheet form of a data.frame, we’d like to use R to get more information about it. In this case, our <code>data.frame</code> is so small that we can print the whole thing and inspect it:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">continents</code></pre></div>
<pre class="output"><code> continent area_km2 population percent_total_pop
1 Africa 30370000 1022234000 15.0
2 Americas 42330000 934611000 14.0
3 Antarctica 13720000 4490 0.0
4 Asia 43820000 4164252000 60.0
5 Europe 10180000 738199000 11.0
6 Oceania 9008500 29127000 0.4
</code></pre>
<p>When we start working with more-realistic datasets, let alone big data, that won’t work. We can get the first few rows of a <code>data.frame</code> with <code>head</code> (in this case all the rows!).</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">head</span>(continents)</code></pre></div>
<pre class="output"><code> continent area_km2 population percent_total_pop
1 Africa 30370000 1022234000 15.0
2 Americas 42330000 934611000 14.0
3 Antarctica 13720000 4490 0.0
4 Asia 43820000 4164252000 60.0
5 Europe 10180000 738199000 11.0
6 Oceania 9008500 29127000 0.4
</code></pre>
<p><code>str</code> provides richer information about a <code>data.frame</code> and each element in it. It is generally a good first-step inspection of an R object.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">str</span>(continents)</code></pre></div>
<pre class="output"><code>'data.frame': 6 obs. of 4 variables:
$ continent : chr "Africa" "Americas" "Antarctica" "Asia" ...
$ area_km2 : num 30370000 42330000 13720000 43820000 10180000 ...
$ population : num 1.02e+09 9.35e+08 4.49e+03 4.16e+09 7.38e+08 ...
$ percent_total_pop: num 15 14 0 60 11 0.4
</code></pre>
<p>We get some summary information on <code>continents</code>: it’s type and dimensions, and we get some information on each variable in the <code>data.frame</code>. We’ll get into the details of variable types later, but for now, note that <code>continent</code> is a character-type variable and the rest are numeric.</p>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h4 id="shoutout"><span class="glyphicon glyphicon-pencil"></span>Shoutout</h4>
</div>
<div class="panel-body">
<p>There is another function like <code>head()</code> and <code>str()</code> that provides information on a <code>data.frame</code>: <code>summary()</code><br />
- Call the <code>summary</code> function on the <code>continent</code> data.<br />
- What is the average (mean) area of a continent?</p>
</div>
</section>
<h3 id="vectors">Vectors</h3>
<p>Vectors are a collection of observations of a single variable, ie a column in a data frame. They are the fundamental unit of R. Let’s explore why that’s useful and how to work with them.</p>
<p>We can extract a vector (ie a single variable) from a data frame with the <code>$</code> operator.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">continents$population</code></pre></div>
<pre class="output"><code>[1] 1022234000 934611000 4490 4164252000 738199000 29127000
</code></pre>
<p>Note that you can use tab-completion to see what variables are available.</p>
<p>That just printed the six values of population. We are going to work with them some more, so extract them from the data.frame and store them to a new object, called <code>pop</code>:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop <-<span class="st"> </span>continents$population</code></pre></div>
<p>Now we have a new object in our environment: a numeric “vector” with six entries. R is built around vectors. In data analysis and statistics, we don’t often work with individual numbers, but multiple observations. This is baked into R and helps it give it its power.</p>
<p>Another core R concept is the idea that when you manipulate an object, the original object doesn’t change. Note that the <code>continents</code> data.frame still has the population variable. R didn’t “take it out” of <code>continents</code>; instead, it made a copy of it and stored it to a variable called <code>pop</code>. They are now two separate things. If we make a change to one, it will not affect the other. There is a powerful computational paradigm here that may not be apparent yet, but keep in mind that anything you do in R, if you want to keep the results, you need to assign them to a new object.</p>
<h4 id="vectorization">Vectorization</h4>
<p>We just said that R is built around vectors. Most functions that accept a single value can also accept a vector of values. E.g. we can find the logarithm of each continent’s population:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">log</span>(<span class="dt">x =</span> pop, <span class="dt">base =</span> <span class="dv">10</span>)</code></pre></div>
<pre class="output"><code>[1] 9.009550 8.970631 3.652246 9.619537 8.868173 7.464296
</code></pre>
<p>R went over each item in <code>pop</code> and took the base-10 logarithm. Some functions take a vector but rather than returning a result for each item, operate on all of them together. E.g. we can find the mean and standard deviation of populations:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">mean</span>(pop)</code></pre></div>
<pre class="output"><code>[1] 1148071248
</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">sd</span>(pop)</code></pre></div>
<pre class="output"><code>[1] 1542519717
</code></pre>
<p>Many functions will also operate element-wise over multiple vectors. E.g. to calculate the population density of each continent, we need to divide the population by the land area for each continent. We can do that with a single command:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">continents$population /<span class="st"> </span>continents$area_km2</code></pre></div>
<pre class="output"><code>[1] 3.365933e+01 2.207916e+01 3.272595e-04 9.503085e+01 7.251464e+01
[6] 3.233280e+00
</code></pre>
<p>Note that we didn’t have to take those vectors out of the data.frame to use them. We can do vectorized operations right in the data.frame, which helps keep everything organized: recall that each row of a data.frame is a particular observation (in this case a continent), so we often want to do operations with each row.</p>
<p>Just like we can extract a column from a data.frame with <code>$</code>, we can make a new column:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">continents$pop_density <-<span class="st"> </span>continents$population /<span class="st"> </span>continents$area_km2
continents</code></pre></div>
<pre class="output"><code> continent area_km2 population percent_total_pop pop_density
1 Africa 30370000 1022234000 15.0 3.365933e+01
2 Americas 42330000 934611000 14.0 2.207916e+01
3 Antarctica 13720000 4490 0.0 3.272595e-04
4 Asia 43820000 4164252000 60.0 9.503085e+01
5 Europe 10180000 738199000 11.0 7.251464e+01
6 Oceania 9008500 29127000 0.4 3.233280e+00
</code></pre>
<h3 id="creating-vectors">Creating vectors</h3>
<p>We can also manually create new vectors. There are many ways to do this, but the most flexible is the combine function, <code>c()</code>.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">3</span>, <span class="dv">5</span>)</code></pre></div>
<pre class="output"><code>[1] 1 3 5
</code></pre>
<p>Let’s store that vector in a new variable. We will use it again soon.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">odds <-<span class="st"> </span><span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">3</span>, <span class="dv">5</span>)</code></pre></div>
<h3 id="subsetting">Subsetting</h3>
<h4 id="subset-by-index">Subset by index</h4>
<p>We can extract items from a vector by specifying which positions, or indices, we want. R’s syntax for subsetting is square brackets (<code>[ ]</code>) at the end of an object containing the positions to return. So to get the third element out of our <code>pop</code> vector:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop</code></pre></div>
<pre class="output"><code>[1] 1022234000 934611000 4490 4164252000 738199000 29127000
</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[<span class="dv">3</span>]</code></pre></div>
<pre class="output"><code>[1] 4490
</code></pre>
<p>To get the first three elements, we need to put a vector containing 1, 2, and 3 inside the <code>[ ]</code>. We just learned how to make such a vector using the combine function.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[<span class="kw">c</span>(<span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>)]</code></pre></div>
<pre class="output"><code>[1] 1022234000 934611000 4490
</code></pre>
<p>Sometimes it will be more useful to provide the desired positions as a variable. Let’s pull out the odd-positioned entries from <code>pop</code>:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[odds]</code></pre></div>
<pre class="output"><code>[1] 1022234000 4490 738199000
</code></pre>
<p>We can also tell R which elements we <em>don’t</em> want with a <code>-</code>. This returns each element in <code>pop</code> except the first one:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[-<span class="dv">1</span>]</code></pre></div>
<pre class="output"><code>[1] 934611000 4490 4164252000 738199000 29127000
</code></pre>
<p>If we try to ask for an element that doesn’t exist, R returns <code>NA</code>. <code>NA</code> is a special value in R that means “missing”.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[<span class="dv">10</span>]</code></pre></div>
<pre class="output"><code>[1] NA
</code></pre>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h4 id="challenge-create-and-subset-a-vector"><span class="glyphicon glyphicon-pencil"></span>Challenge – Create and subset a vector</h4>
</div>
<div class="panel-body">
<p>Similar to <code>c</code>, the <code>seq</code> function creates a vector: a <strong>seq</strong>uence of numbers.</p>
<ul>
<li><p>Your first task is to create a sequence of all the multiples of three from three to 300. Figure out how to do this. Some combination of playing with the function in the console and reading its helpfile (<code>?seq</code>) should work. Helpfiles are challenging at first, but it’s important to learn how to find the info you need in them. Hint: The arguments you need here are <code>from</code>, <code>to</code>, and <code>by</code>.</p></li>
<li>Store your sequence in a variable.</li>
<li>Extract the 33rd entry from the sequence</li>
<li><strong>Bonus</strong>: Create a vector with ten evenly-spaced numbers starting with one and ending with one-million. What is the sum of the second and ninth entries in this vector?</li>
<li><p><strong>Super-Bonus</strong>: Returning to the multiples-of-three vector, what is the sum of the numbers in positions that are not evenly divisible by three? That is, the sum of the first, second, fourth, fifth, seventh, etc. entries.</p></li>
</ul>
</div>
</section>
<h4 id="subset-by-logical">Subset by logical</h4>
<p>Note that the <code>continents</code> data frame contains two common kinds of variables: numbers (num) and strings (chr).</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">str</span>(continents)</code></pre></div>
<pre class="output"><code>'data.frame': 6 obs. of 5 variables:
$ continent : chr "Africa" "Americas" "Antarctica" "Asia" ...
$ area_km2 : num 30370000 42330000 13720000 43820000 10180000 ...
$ population : num 1.02e+09 9.35e+08 4.49e+03 4.16e+09 7.38e+08 ...
$ percent_total_pop: num 15 14 0 60 11 0.4
$ pop_density : num 3.37e+01 2.21e+01 3.27e-04 9.50e+01 7.25e+01 ...
</code></pre>
<p>A third important data type in R is logicals. You saw this when making logical comparisons like <code>1 > 0</code>. Logical data can only be <code>TRUE</code> or <code>FALSE</code> (or <code>NA</code> for missing; any variable type in R can have <code>NA</code>-missing values).</p>
<p>Logical comparisons are vectorized like other things in R. Let’s find the highly populated continents, those with more than one-billion people. One-billion is 10^9, so we can write it as <code>1e9</code>.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop ><span class="st"> </span><span class="fl">1e9</span></code></pre></div>
<pre class="output"><code>[1] TRUE FALSE FALSE TRUE FALSE FALSE
</code></pre>
<p>That went over each element in <code>pop</code> and compared it with 10. We say that R “recycled” 10 to compare it with each element in <code>pop</code>.</p>
<p>In the same way that we subset by index before, we can subset by a logical vector. To find the values of <code>pop</code> that were greater than one-billion, we subset it like so:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">pop[pop ><span class="st"> </span><span class="fl">1e9</span>]</code></pre></div>
<pre class="output"><code>[1] 1022234000 4164252000
</code></pre>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h4 id="shoutout-1"><span class="glyphicon glyphicon-pencil"></span>Shoutout</h4>
</div>
<div class="panel-body">
<p>How could you extract the names of the continents with more than one-billion people?</p>
</div>
</section>
<p>Recall that in a data frame, each row constitutes a single observation. This makes it especially useful to test one column and use it to subset another – we often want the entries of some variable where a condition on another variable is met. For example, to find the land-area of Africa, we can test the continent names for being “Africa” and extract the area where that is true.</p>
<p>Here as elsewhere, it is often useful to build code from the insight out, ie, write the logical test first, then go left and write what you want to subset with it.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">continents$area_km2[continents$continent ==<span class="st"> "Africa"</span>]</code></pre></div>
<pre class="output"><code>[1] 30370000
</code></pre>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h4 id="mcq-subset-and-vectorize"><span class="glyphicon glyphicon-pencil"></span>MCQ – Subset and vectorize</h4>
</div>
<div class="panel-body">
<p>What is the total land area of continents that have at least 10% of the world’s population?</p>
<ul>
<li>Use subsetting to get the areas you want</li>
<li>Use the <code>sum</code> function to find the total land area</li>
</ul>
<ol style="list-style-type: lower-alpha">
<li>149428500</li>
<li>126700000</li>
<li>22728500</li>
<li>100</li>
</ol>
</div>
</section>
<h3 id="reading-csv-data">Reading csv data</h3>
<p>The <code>continents</code> data.frame was useful for learning because it was so small, but it’s time to graduate to something more interesting and realistic. Data come in many forms, and we need to be able to load them in R. For our own use and with others who use R, there are R-specific data structures we can use, like the .RDA file-type we just saw, but we need to be able to work with more general data types too. Comma-separated value (csv) tables are perhaps the most universal data structure.</p>
<p>The gapminder dataset provides country-by-year level data on gdp, population, and longevity. I downloaded it and put it in the data directory of my project. You will do the same in a minute.</p>
<p>We can read csv’s with the <code>read.csv</code> function. The argument to <code>read.csv</code> is the location of the file, relative to your working directory. Since I saved the gapminder data to the <code>data</code> directory of my project, I can load it with this. Note the use of tab-completion to find the file and get it exactly right without typos.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">read.csv</span>(<span class="st">'data/gapminder-FiveYearData.csv'</span>)</code></pre></div>
<p>Whoa! What just happened? R executed the function and printed the result, just like when you enter <code>log(1)</code>. How do you store an object to a variable?</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">gapminder <-<span class="st"> </span><span class="kw">read.csv</span>(<span class="st">'data/gapminder-FiveYearData.csv'</span>)</code></pre></div>
<p>Now, a data.frame called <code>gapminder</code> is in my Environment, and I can see that it is a data.frame with 1704 rows and 6 columns.</p>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h4 id="challenge-read-csv-data"><span class="glyphicon glyphicon-pencil"></span>Challenge – read csv data</h4>
</div>
<div class="panel-body">
<p>The gapminder data are available at <a href="https://raw.githubusercontent.com/resbaz/r-novice-gapminder-files/master/data/gapminder-FiveYearData.csv">this link</a>.<br />
- Right click on the link to “save file as…”<br />
- Save the .csv file in the <code>/data</code> directory of your project.<br />
- Read the data with the <code>read.csv</code> function and assign it to the variable <code>gapminder</code>.<br />
- Inspect the data.frame using the <code>summary</code> function. What is the most recent year for which we have data?</p>
<p><strong>Advanced challenge</strong></p>
<p>Suppose you get a .csv file from a colleague in Europe. Because they use “,” (comma) as a decimal separator, they use “;” (semi-colons) to separate fields. How can you read it into R?</p>
<p>Feel free to use the web and/or R’s helpfiles.</p>
</div>
</section>
</div>
</div>
</article>
<div class="footer">
<a class="label swc-blue-bg" href="http://software-carpentry.org">Software Carpentry</a>
<a class="label swc-blue-bg" href="https://github.com/swcarpentry/lesson-template">Source</a>
<a class="label swc-blue-bg" href="mailto:[email protected]">Contact</a>
<a class="label swc-blue-bg" href="LICENSE.html">License</a>
</div>
</div>
<!-- Javascript placed at the end of the document so the pages load faster -->
<script src="http://software-carpentry.org/v5/js/jquery-1.9.1.min.js"></script>
<script src="css/bootstrap/bootstrap-js/bootstrap.js"></script>
</body>
</html>