Download R Programming: A Beginner's Guide to Data Manipulation and Analysis and more Study notes Advanced Computer Programming in PDF only on Docsity!
Transform and Tidy (Wrangle) Data with R - Required
Read
Bojan Duric
This Notebook is selection of “A Very (short) Introduction to R” by Paul Torfs & Claudia Brauer and “R for Data Scince”" by Hadley Wickham and Garrett Grolemund
1 Introduction
What is R?
R is a powerful language and environment for statistical computing and graphics. It is a public domain (a so called “GNU”) project which is similar to the commercial S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S, or in language terms different dialect of S. The main advantages of R are the fact that R is freeware and that there is a lot of help available online. It is quite similar to other programming packages such as MatLab (not freeware), but more user-friendly than programming languages such as C++ or Fortran. You can use R as it is, but for educational purposes we prefer to use R in combination with the RStudio interface (also freeware), which has an organized layout and several extra options. The R language came to use quite a bit after S had been developed. One key limitation of the S language was that it was only available in a commercial package, S-PLUS. In 1991, R was created by Ross Ihaka and Robert Gentleman in the Department of Statistics at the University of Auckland. In 1993 the first announcement of R was made to the public. In 1995, Martin Mächler made an important contribution by convincing Ross and Robert to use the GNU General Public License to make R free software. This was critical because it allowed for the source code for the entire R system to be accessible to anyone who wanted to tinker with it (more on free software later). In 1996, a public mailing list was created (the R-help and R-devel lists) and in 1997 the R Core Group was formed, containing some people associated with S and S-PLUS. Currently, the core group controls the source code for R and is solely able to check in changes to the main R source tree. Finally, in 2000 R version 1.0. was released to the public.
Limitations of R
No programming language or statistical analysis system is perfect. R certainly has a number of drawbacks. For starters, R is essentially based on almost 50 year old technology, going back to the original S system developed at Bell Labs. There was originally little built in support for dynamic or 3-D graphics (but things have improved greatly since the “old days”). Another commonly cited limitation of R is that objects must generally be stored in physical memory. This is in part due to the scoping rules of the language, but R generally is more of a memory hog than other statistical packages. However, there have been a number of advancements to deal with this, both in the R core and also in a number of packages developed by contributors. Also, computing power and capacity has continued to grow over time and amount of physical memory that can be installed on even a consumer-level laptop is substantial. While we will likely never have enough physical memory on a computer to handle the increasingly large datasets that are being generated, the situation has gotten quite a bit easier over time.
At a higher level one “limitation” of R is that its functionality is based on consumer demand and (voluntary) user contributions. If no one feels like implementing your favorite method, then it’s your job to implement it (or you need to pay someone to do it). The capabilities of the R system generally reflect the interests of the R user community. As the community has ballooned in size over the past 10 years, the capabilities have similarly increased.
2 Getting started
2.1 Install R
To install R on your computer (legally for free!), go to the home website of R: http://www.r-project.org/ and do the following (assuming you work on a windows computer):
- click download CRAN in the left bar
- choose a download site
- choose Windows as target operation system
- click base
- choose Download R 3.** for Windows and
- choose default answers for all questions
It is also possible to run R and RStudio from a USB stick instead of installing them. This could be useful when you don’t have administrator rights on your computer. See our separate note “How to use portable versions of R and RStudio” for help on this topic.
2.2 Install RStudio
After finishing this setup, you should see an “R” icon on you desktop. Clicking on this would start up the standard interface. We recommend, however, to use the RStudio interface. To install RStudio, go to: http://www.rstudio.org/ and do the following (assuming you work on a windows computer):
- click Download RStudio
- click Download RStudio Desktop
- click Recommended For Your System
- download the .exe file and run it (choose default answers for all questions)
2.3 RStudio layout
The RStudio interface consists of several windows (see Figure 1).
- Bottom left: console window (also called command window). Here you can type simple commands after the “>” prompt and R will then execute your command. This is the most important window, because this is where R actually does stuff.
- Top left: editor window (also called script window). Collections of commands (scripts) can be edited and saved. When you don’t get this window, you can open it with File -> New -> R script Just typing a command in the editor window is not enough, it has to get into the command window before R executes the command. If you want to run a line from the script window (or the whole script), you can click Run or press CTRL+ENTER to send it to the command window.
- Top right: workspace / history window. In the workspace window you can see which data and values R has in its memory. You can view and edit the values by clicking on them. The history window shows what has been typed before.
There are many more packages available on the R website. If you want to install and use a package (for example, the package called “dplyr”) you should:
- Install the package: click install packages in the packages window and type geometry or type in- stall.packages(“dplyr”) in the command window. # install.packages("dplyr")
- Load the package: check box in front of geometry or type library(“dplyr”) in the command window.
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
3 Some first examples of R commands
This is section is dedicated to basic R use and syntax. Please do read this section to understand simple R syntax and possible data formats and types. It is highly recommended to complete some of the swirl() exercises.
3.1 Calculator
R can be used as a calculator. You can just type your equation in the command window after the “>” or editor: 10^2 + 36
[1] 136
and R will give the answer
Practice
Compute the difference between 2014 and the year you started at this university and divide this by the difference between 2014 and the year you were born. Multiply this with 100 to get the percentage of your life you have spent at this university. Use brackets if you need them. If you use brackets and forget to add the closing bracket, the “>” on the command line changes into a “+”. The “+” can also mean that R is still busy with some heavy computation. If you want R to quit what it was doing and give back the “>”, press ESC (see the reference list on the last page).
_# this is comment
use hastag to comment your code
write your code:_
3.2 Workspace
You can also give numbers a name. By doing so, they become so-called variables which can be used later. For example, you can type in the command window:
a <- 4
You can see that a appears in the workspace window, which means that R now remembers what a is. You can also ask R what a is (just type a ENTER in the command window): a
[1] 4
# or do calculations with a:
a * 5
[1] 20
# You can also assign a new value to a using the old one. a <- a + 10 a
[1] 14
To remove all variables from R’s memory, type
rm (list= ls ())
or click “clear all” in the workspace window. You can see that RStudio then empties the workspace window. If you only want to remove the variable a, you can type rm(a) or change list to grid in the workspace widow and check variable/object you want to remove.
3.3 Scalars, vectors and matrices
Like in many other programs, R organizes numbers in scalars (a single number - 0-dimensional), vectors (a row of numbers, also called arrays - 1-dimensional) and matrices (like a table - 2- dimensional). The a you defined before was a scalar. To define a vector with the numbers 3, 4 and 5, you need the function c, which is short for concatenate (paste together).
b <- c (3,4,5)
Matrices and other 2-dimensional structures will be introduced in Section 5.
How would you use what you learned to install more than one R package in one command line
# write your code:
Swirl Practice
In R console run following commands:
- library(swirl)
- install_course_github(“swirldev”, “R_Programming_E”)
- swirl()
Select 1 for ‘R Programming’ then Select 1 for ‘Basic Building Blocks’. At the end of exercise select 0 to exit. Hint: R console will display exercise completion %.
3.4 Functions
If you would like to compute the mean of all the elements in the vector b from the example above, you could type
Index
x
- In the first line, 100 random numbers are assigned to the variable x, which becomes a vector by this operation.
- In the second line, all these values are plotted in the plots window.
4 Scripts
R is an interpreter that uses a command line based environment. This means that you have to type commands, rather than use the mouse and menus. This has the advantage that your tasks can be automated.
You can store your commands in files, the so called scripts. These scripts have typically file names with the extension .R, e.g. foo.R. You can open an editor window to edit these files by clicking File and New or Open file...
You can run (send to the console window) part of the code by selecting lines and pressing CTRL+ENTER or click Run in the editor window. If you do not select anything, R will run the line your cursor is on. You can always run the whole script with the console command source, so e.g. for the script in the file foo.R you type: source(“foo.R”)
You can also click Run all in the editor window or type CTRL+SHIFT + S to run the whole script at once.
Practice (optional)
Make a file called firstscript.R containing Rcode that generates 100 random numbers and plots them, and run this script several times.
5 Data structures
If you are unfamiliar with R, it makes sense to just retype the commands listed in this section. Maybe you will not need all these structures in the beginning, but it is always good to have at least a first glimpse of the terminology and possible applications.
5.1 Vectors
Vectors were already introduced, but they can do more:
vec1 <- c (1,4,6,8,10) vec
[1] 1 4 6 8 10
vec1[5]
[1] 10
vec1[3] <- 12 vec
[1] 1 4 12 8 10
vec2 <- seq (from=0, to=1, by=0.25) vec
[1] 0.00 0.25 0.50 0.75 1.
sum (vec1)
[1] 35
vec1 + vec
[1] 1.00 4.25 12.50 8.75 11.
Swirl Practice
In R console run following commands:
library (swirl)
| Hi! I see that you have some variables saved in your workspace. To keep
| things running smoothly, I recommend you clean up before starting swirl.
| Type ls() to see a list of the variables in your workspace. Then, type
| rm(list=ls()) to clear your workspace.
| Type swirl() when you are ready to begin.
Select 1 for’R Programming’ then Select 4 for Vectors. At the end of exercise select 0 to exit.
## [1] 8.
Practice (optional)
Make a script file which constructs three random normal vectors of length 100. Call these vectors x1, x2 and x3. Make a data frame called t with three columns (called a, b and c) containing respectively x1, x1+x and x1+x2+x3. Call the following functions for this data frame: plot(t) and sd(t). Can you understand the results? Rerun this script a few times.
# write your code:
Swirl Practice (highly Recommended)
In R console run following commands:
Select 1 for’R Programming’ then Select 7 for Matrices and Data Frames. At the end of exercise select 0 to exit.
5.4 Lists
Another basic structure in R is a list. The main advantage of lists is that the “columns” (they’re not really ordered in columns any more, but are more a collection of vectors) don’t have to be of the same length, unlike matrices and data frames.
L <- list (one=1, two= c (1,2), five= seq (0, 1, length=5)) L
$one
[1] 1
$two
[1] 1 2
$five
[1] 0.00 0.25 0.50 0.75 1.
L$one
[1] 1
L$two
[1] 1 2
L$five
[1] 0.00 0.25 0.50 0.75 1.
names (L)
[1] "one" "two" "five"
L$five + 10
[1] 10.00 10.25 10.50 10.75 11.
6 Graphics
Plotting is an important statistical activity. So it should not come as a surprise that R has many plotting facilities. The following lines show a simple plot:
plot ( rnorm (100), type="l", col="gold")
Index
rnorm(100)
_#ToDo
check help for plot and change some arguments
re-run_
Hundred random numbers are plotted by connecting the points by lines (the symbol between quotes after the type=, is the letter l stands for lines, not the number 1) in a gold color.
Another very simple example is the classical statistical histogram plot, generated by the simple command:
hist ( rnorm (100))
Index
t$x
# Try to find out, either by experimenting or by using the help, what the meaning is of rgb, the last ar
Swirl Practice (optional)
In R console run following commands:
Select 1 for ‘R Programming’ then Select 15 for Base Graphics. At the end of exercise select 0 to exit.
7 Reading and writing data files
There are many ways to write data from within the R environment to files, and to read data from files.
The following lines illustrate the essential:
d <- data.frame (a = c (3,4,5), b = c (12,43,54)) d
a b
1 3 12
2 4 43
3 5 54
write.table (d, file="tst0.txt", row.names=FALSE)
d2 <- read.table (file="tst0.txt", header=TRUE) d
a b
1 3 12
2 4 43
3 5 54
- In line 1, a simple example data frame is constructed and stored in the variable d.
- Line 2 shows the content of this data frame: two columns (called a and b), each containing three numbers.
- Line 3 writes this data frame to a text file, called tst0.txt in working directory. (check your working directory by getwd() ) The argument row.names=FALSE prevents that row names are written to the file. Because nothing is specified about col.names, the default option col.names=TRUE is chosen and column names are written to the file.
Practice
Make a file called tst1.txt in Notepad a g x 1 2 3 4 5 6 7 8 9 10 11 12 and store it in your working directory. Write a script to read it, to multiply the column called g by 5 and to store it as tst2.txt. # write your code:
8 Not available data
Practice (optional)
Compute the mean of the square root of a vector of 100 random numbers. What happens? # write your code:
When you work with real data, you will encounter missing values because instrumentation failed or because you didn’t want to measure in the weekend. When a data point is not available, you write NA instead of a number. j <- c (1,2,NA)
Computing statistics of incomplete data sets is strictly speaking not possible. Maybe the largest value occurred during the weekend when you didn’t measure. Therefore, R will say that it doesn’t know what the largest value of j is: max (j)
[1] NA
If you don’t mind about the missing data and want to compute the statistics anyway, you can add the argument na.rm=TRUE (Should I remove the NAs? Yes!). max (j, na.rm=TRUE)
[1] 2
10.1 If-statement
The if-statement is used when certain computations should only be done when a certain condition is met (and maybe something else should be done when the condition is not met). An example:
w <- 3 if( w < 5 ){ d<- }else{ d<- } d
[1] 2
- In line 2 a condition is specified: w should be less than 5. *If the condition is met, R will execute what is between the first brackets in line 3.
- If the condition is not met, R will execute what is between the second brackets, after the else in line 4. You can leave the else{... }-part out if you don’t need it.
- In this case, the condition is met and d has been assigned the value 2. To get a subset of points in a vector for which a certain condition holds, you can use a shorter method: a <- c (1,2,3,4) b <- c (5,6,7,8) f <- a[b==5 | b==8] f
[1] 1 4
- In line 1 and 2 two vectors are made.
- In line 3 you say that f is composed of those elements of vector a for which b equals 5 or b equals 8. Note the double = in the condition. Other conditions (also called logical or Boolean operators) are <, >, != (6=), <= (???) and >= (???). To test more than one condition in one if-statement, use & if both conditions have to be met (“and”) and | if one of the conditions has to be met (“or”).
10.2 For-loop (optional)
If you want to model a time series, you usually do the computations for one time step and then for the next and the next, etc. Because nobody wants to type the same commands over and over again, these computations are automated in for-loops. In a for-loop you specify what has to be done and how many times. To tell “how many times”, you specify a so-called counter. An example: h <- seq (from=1, to=8) s <- c () for(i in 2:10){ s[i] <- h[i] * 10 } s
[1] NA 20 30 40 50 60 70 80 NA NA
- First the vector h is made.
- In line 2 an empty vector s is created. This is necessary because when you introduce a variable within the for-loop, R will not remember it when it has gotten out of the for-loop.
- In line 3 the for-loop starts. In this case, i is the counter and runs from 2 to 10.
- Everything between the curly brackets is processed 9 times. The first time i=2, the second element of h is multiplied with 10 and placed in the second position of the vector s. The second time i=3, etc. In the last two runs, the 9th and 10th elements of h are requested, which do not exist. Note that these statements are evaluated without any explicit error messages.
Practice (optional)
Make a vector from 1 to 100. Make a for-loop which runs through the whole vector. Multiply the elements which are smaller than 5 and larger than 90 with 10 and the other elements with 0.1.
# write your code:
10.3 Writing your own functions (optional)
Functions you program yourself work in the same way as pre-programmed R functions.
fun1 <- function(arg1, arg2 ) { w = arg1 ^ 2 return (arg2 + w) }
fun1 (arg1 = 3, arg2 = 5)
[1] 14
fun1 (3, 5)
[1] 14
- In line 1 the function name (fun1) and its arguments (arg1 and arg2) are defined.
- Lines 2-5 specify what the function should do if it is called. The return value (arg2+w) is shown on the screen.
- In line 6-7 the function is called with arguments 3 and 5.
11 Some useful references
This is a collection of very useful R base functions
11.1 R Functions
This is a subset of the functions explained in the R reference card.
Data creation
- read.table: read a table from file. Arguments: header=TRUE: read first line as titles of the columns; sep=“,”: numbers are separated by commas; skip=n: don’t read the first n lines.
- write.table: write a table to file
- c: paste numbers together to create a vector
- array: create a vector, Arguments: dim: length
- matrix: create a matrix, Arguments: ncol and/or nrow: number of rows/columns
- data.frame: create a data frame
- paste: paste character strings together
- substr: extract part of a character string
Fitting
- lm(v1???v2): linear fit (regression line) between vector v1 on the y-axis and v2 on the x-axis
- nls(v1???a+b*v2, start=ls(a=1,b=0)): nonlinear fit. Should contain equation with variables (here v and v2 and parameters (here a and b) with starting values
- coef: returns coefficients from a fit
- summary: returns all results from a fit
Plotting
- plot(x): plot x (y-axis) versus index number (x-axis) in a new window
- plot(x,y): plot y (y-axis) versus x (x-axis) in a new window
- image(x,y,z): plot z (color scale) versus x (x-axis) and y (y-axis) in a new window
- lines or points: add lines or points to a previous plot
- hist: plot histogram of the numbers in a vector
- barplot: bar plot of vector or data frame
- contour(x,y,z): contour plot
- abline: draw line (segment). Arguments: a,b for intercept a and slope b; or h=y for horizontal line at y; or v=x for vertical line at x.
- curve: add function to plot. Needs to have an x in the expression. Example: curve(xˆ2)
- legend: add legend with given symbols (lty or pch and col) and text (legend) at location (x=“topright”)
- axis: add axis. Arguments: side - 1=bottom, 2=left, 3=top, 4=right
- mtext: add text on axis. Arguments: text (character string) and side
- grid: add grid
- par: plotting parameters to be specified before the plots. Arguments: e.g. mfrow=c(1,3)): number of figures per page (1 row, 3 columns); new=TRUE: draw plot over previous plot.
Plotting parameters
These can be added as arguments to plot, lines, image, etc. For help see par. * type: “l”=lines, “p”=points, etc. * col: color - “blue”, “red”, etc * lty: line type - 1=solid, 2=dashed, etc. * pch: point type - 1=circle, 2=triangle, etc. * main: title - character string * xlab and ylab: axis labels - character string * xlim and ylim: range of axes - e.g. c(1,10) * log: logarithmic axis - “x”, “y” or “xy”
Programming
- function(arglist){expr}: function definition: do expr with list of arguments arglist
- if(cond){expr1}else{expr2}: if-statement: if cond is true, then expr1, else expr
- for(var in vec) {expr}: for-loop: the counter var runs through the vector vec and does expr each run
- while(cond){expr}: while-loop: while cond is true, do expr each run
11.2 Keyboard shortcuts
In RStudio IDE Alt+Shift+K to see all keyboard shortcuts
11.3 Error messages
- No such file or directory or Cannot change working directory Make sure the working directory and file names are correct.
- Object ‘x’ not found The variable x has not been defined yet. Define x or write apostrophes if x should be a character string.
- Argument ‘x’ is missing without default You didn’t specify the compulsory argument x.
- – R is still busy with something or you forgot closing brackets. Wait, type } or ) or press ESC.
- Unexpected ‘)’ in “)” or Unexpected ‘}’ in “}” The opposite of the previous. You try to close something which hasn’t been opened yet. Add opening brackets.
- Unexpected ‘else’ in “else” Put the else of an if-statement on the same line as the last bracket of the “then”-part: }else{.
- Missing value where TRUE/FALSE needed Something goes wrong in the condition-part (if(x==1)) of an if-statement. Is x NA?
- The condition has length > 1 and only the first element will be used In the condition-part (if(x==1)) of an if statement, a vector is compared with a scalar. Is x a vector? Did you mean x[i]?
- Non-numeric argument to binary operator You are trying to do computations with something which is not a number. Use class(... ) to find out what went wrong or use as.numeric(... ) to transform the variable to a number.
- Argument is of length zero or Replacement is of length zero The variable in question is NULL, which means that it is empty, for example created by c(). Check the definition of the variable.
Tidy Data with R Bootcamp Introduction
Data science is an exciting and growing discipline that allows you to turn raw data into understanding, insight, and knowledge. The goal of “Explore and Tidy Your Data” is to help you learn the most important tools in R that will allow you to do data science. After the bootcamp, you’ll have the tools to tackle a wide variety of data science challenges where additional steps to prepare your data for analysis are required and repetitive, using the best parts of R.
What you will learn
The goal is to give you a solid foundation in the most important data manipulation R tools. Our model of the tools needed in a typical data science project looks something like this:
- First you must import your data into R. This typically means that you take data stored in a file, database, or web API, and load it into a data frame in R. If you can’t get your data into R, you can’t do data science on it!
- Once you’ve imported your data, it is a good idea to tidy it. Tidying your data means storing it in a consistent form that matches the semantics of the dataset with the way it is stored. In brief, when your data is tidy, each column is a variable, and each row is an observation. Tidy data is important because the consistent structure lets you focus your struggle on questions about the data, not fighting to get the data into the right form for different functions.
- With tidy data, a common first step is to transform it. Transformation includes narrowing in on observations of interest (like all people in one city, or all data from the last year), creating new variables that are functions of existing variables (like computing velocity from speed and time), and calculating a set of summary statistics (like counts or means). Tidying and transforming are called wrangle.
