Dynamic Documents

• I write these slides using a mix of markdown, R, and LaTeX.
• Behind the scenes, pandoc and knitr dynamically convert everything into reveal.js slides.
• I could just have easily generated beamer slides like these ones or a LaTeX Document like this one.
• If I were less virtuous, I could even have generated a Microsoft Word Document!
• I can do all this, and more, with hardly any effort using the wonderful tools Quarto and R Markdown.

This is a Slide

• Markdown is fun! You can write in italics or even in boldface!
• Including links is easy too.

\[ \int_{-\infty}^\infty f(x)\, dx = 1 \]

Look it’s some R code!

message <- 'hello world'
print(message)

[1] "hello world"

Quarto versus R Markdown

• They are extremely similar.
• Quarto is a newer and more powerful, and has first-class support for languages besides R, including Julia and Python
• R Markdown is older, better-documented, and less buggy.
• Code chunks, Markdown, and LaTeX work the same.
• Configuration details are a bit different.
• Use whichever you prefer
• Note that Quarto supports emoji 😍

💪 Exercise (\(\infty\) minutes)

Create a new R Markdown or Quarto document: your choice. I suggest .html output. Use this document to take notes and complete exercises in all future lectures!

How should you store your data?

• Always use flat text, .csv or .tsv, unless you have extremely specialized needs (e.g. GIS)
• Never use a proprietary binary format like STATA’s .dta
• Never use Excel

👼 Loading the data of the virtuous

• Some angelic beings store their data in a reasonable way!
• readr package loads .csv, .tsv and related files.
• readr is in tidyverse so you already installed it!
• readr functions are superior to similar base R functions
• References:
  • Data Import Chapter from R for Data Science
  • readr cheat sheet
  • readr website

👹 Hell is other people’s data.

tidyverse has two helpful packages for importing the data of damned souls who writhe in eternal torment:

• readxl imports .xls and xlsx files
  • readxl website
  • readxl cheatsheet
• haven imports SAS, SPSS, and Stata files
  • haven website

“Core” Tidyverse Packages

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.2     ✔ readr     2.1.4
✔ forcats   1.0.0     ✔ stringr   1.5.0
✔ ggplot2   3.4.2     ✔ tibble    3.2.1
✔ lubridate 1.9.2     ✔ tidyr     1.3.0
✔ purrr     1.0.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

• This shows you which tidyverse pacakges are loaded automatically when you run library(tidyverse).
• Some others, e.g. haven have to be loaded individually.

Example: read_csv()

url <- 'https://ditraglia.com/data/height-handspan.csv'
height_handspan <- read_csv(url)

height_handspan

# A tibble: 326 × 2
   height handspan
    <dbl>    <dbl>
 1     73     22.5
 2     65     17  
 3     69     21  
 4     71     25.5
 5     78     25  
 6     68     20  
 7     75     20.5
 8     72     24.5
 9     63     18  
10     67     21.5
# ℹ 316 more rows

Getting and Setting File Paths

• getwd() returns the current working directory
• setwd() changes the current working directory
• On Windows you have two options:
  1. Replace backslashes with forward slashes: "C:/Users/username/Documents/data.csv"
  2. Use double backslashes: "C:\\Users\\username\\Documents\\data.csv"
• Better plan: use R Studio Projects and relative paths.

💪 Exercise A - (8 min)

1. Download https://ditraglia.com/data/STAR.csv and save this file on your local machine. Then load it with read_csv(). Note that this will require you to specify the path to this file on your local machine.
2. The file final5.dta from the Angrist data archive contains data from the article “Using Maimonides Rule to estimate the Effect of Class Size on Student Achievement” by Angrist & Lavy. Locate and download this file. Then try to load it with read_dta(). You may get an error. If so, see the section “Character encoding” in the associated R help file and follow the instructions given there.

gradebook

library(tidyverse)
set.seed(92815)
gradebook <- tibble(
  student_id = c(192297, 291857, 500286, 449192, 372152, 627561), 
  name = c('Alice', 'Bob', 'Charlotte', 'Dante', 
           'Ethelburga', 'Felix'),
  quiz1 = round(rnorm(6, 65, 15)),
  quiz2 = round(rnorm(6, 88, 5)),
  quiz3 = round(rnorm(6, 75, 10)),
  midterm1 = round(rnorm(6, 75, 10)),
  midterm2 = round(rnorm(6, 80, 8)), 
  final = round(rnorm(6, 78, 11)))
gradebook

# A tibble: 6 × 8
  student_id name       quiz1 quiz2 quiz3 midterm1 midterm2 final
       <dbl> <chr>      <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl>
1     192297 Alice         64    96    68       81       90    99
2     291857 Bob           58    91    91       75       75    79
3     500286 Charlotte     70    94    71       81       70    74
4     449192 Dante         57    85    84       83       94    83
5     372152 Ethelburga    74    91    70       63       73    96
6     627561 Felix         77    86    68       78       83    75

tidyselect in dplyr

• So far used select() by writing column names in full
• This is cumbersome when working with large and complicated tibbles.
• tidyselect provides a helpful collection of functions and operators to make selection easier, faster and more flexible.
• For more details see the documentation.

tidyselect with gradebook

Selecting the quiz columns:

gradebook |> 
  select(quiz1, quiz2, quiz3)

# A tibble: 6 × 3
  quiz1 quiz2 quiz3
  <dbl> <dbl> <dbl>
1    64    96    68
2    58    91    91
3    70    94    71
4    57    85    84
5    74    91    70
6    77    86    68

• Easy enough in this case, but once I taught a course that had eleven quizzes!
• Tedious to type them all out, and easy to make a mistake and forget one.

starts_with()

Select columns based on a common prefix

gradebook |> 
  select(starts_with('quiz'))

# A tibble: 6 × 3
  quiz1 quiz2 quiz3
  <dbl> <dbl> <dbl>
1    64    96    68
2    58    91    91
3    70    94    71
4    57    85    84
5    74    91    70
6    77    86    68

ends_with()

Select columns based on a common suffix

gradebook |> 
  select(ends_with('2'))

# A tibble: 6 × 2
  quiz2 midterm2
  <dbl>    <dbl>
1    96       90
2    91       75
3    94       70
4    85       94
5    91       73
6    86       83

contains()

Select columns that contain a particular string

gradebook |> 
  select(contains('iz'))

# A tibble: 6 × 3
  quiz1 quiz2 quiz3
  <dbl> <dbl> <dbl>
1    64    96    68
2    58    91    91
3    70    94    71
4    57    85    84
5    74    91    70
6    77    86    68

num_range()

Select based on both a prefix and a numeric range

gradebook |> 
  select(num_range('quiz', 2:3))

# A tibble: 6 × 2
  quiz2 quiz3
  <dbl> <dbl>
1    96    68
2    91    91
3    94    71
4    85    84
5    91    70
6    86    68

where()

Accepts a function as input; applies to every column of the tibble and returns those where the function returns TRUE

gradebook |> 
  select(where(is.numeric))

# A tibble: 6 × 7
  student_id quiz1 quiz2 quiz3 midterm1 midterm2 final
       <dbl> <dbl> <dbl> <dbl>    <dbl>    <dbl> <dbl>
1     192297    64    96    68       81       90    99
2     291857    58    91    91       75       75    79
3     500286    70    94    71       81       70    74
4     449192    57    85    84       83       94    83
5     372152    74    91    70       63       73    96
6     627561    77    86    68       78       83    75

Computing the average on each quiz

It’s easy to compute the average on each quiz:

gradebook |> 
  summarize(quiz1_avg = mean(quiz1),  
            quiz2_avg = mean(quiz2),  
            quiz3_avg = mean(quiz3)) 

# A tibble: 1 × 3
  quiz1_avg quiz2_avg quiz3_avg
      <dbl>     <dbl>     <dbl>
1      66.7      90.5      75.3

But would you really want to type this out eleven times in a course with that many quizzes?!

Column-wise Operations: across()

gradebook |> 
  summarize(across(starts_with('quiz'), mean, .names = '{.col}_avg'))

# A tibble: 1 × 3
  quiz1_avg quiz2_avg quiz3_avg
      <dbl>     <dbl>     <dbl>
1      66.7      90.5      75.3

• 1st Argument: .cols specifies columns to work with
  • Specify explicitly e.g. c('col1name', 'col2name')
  • Or use use tidyselect
• 2nd Argument: .fns specifies function(s) to apply
• 3rd Argument (optional): .names sets rule for naming transformed columns, using syntax from glue package

Supplying a Custom Function

zscore <- function(x) {
  (x - mean(x, na.rm = TRUE)) / sd(x, na.rm = TRUE)
}

gradebook |> 
  mutate(across(starts_with('quiz'), zscore, 
                .names = '{.col}_zscore')) |> 
  select(ends_with('zscore'))

# A tibble: 6 × 3
  quiz1_zscore quiz2_zscore quiz3_zscore
         <dbl>        <dbl>        <dbl>
1       -0.320        1.27        -0.752
2       -1.04         0.116        1.61 
3        0.400        0.809       -0.444
4       -1.16        -1.27         0.889
5        0.880        0.116       -0.547
6        1.24        -1.04        -0.752

Supplying a list of functions

mean_var <- list(
  mean = \(x) mean(x, na.rm = TRUE),
  var = \(x) var(x, na.rm = TRUE)
)

gradebook |> 
  summarize(across(starts_with('midterm'), mean_var, 
                   .names = '{.col}_{.fn}'))

# A tibble: 1 × 4
  midterm1_mean midterm1_var midterm2_mean midterm2_var
          <dbl>        <dbl>         <dbl>        <dbl>
1          76.8         53.8          80.8         95.0

Example: STAR Project

• STAR: “Student-Teacher Achievement Ratio”
• Four-year longitudinal study (panel data)
• 11,601 Students in Tennessee from 1985-1989
• Randomly assign students to classes of different sizes.
• Causal effect of class size in early grades?
• Mosteller (1997)

STAR Dataset

STAR Dataset

library(tidyverse)
star <- read_csv('https://ditraglia.com/data/STAR.csv')
star

# A tibble: 6,325 × 6
    race classtype yearssmall hsgrad g4math g4reading
   <dbl>     <dbl>      <dbl>  <dbl>  <dbl>     <dbl>
 1     1         3          0     NA     NA        NA
 2     2         3          0     NA    706       661
 3     1         3          0      1    711       750
 4     2         1          4     NA    672       659
 5     1         2          0     NA     NA        NA
 6     1         3          0     NA     NA        NA
 7     1         1          4     NA    668       657
 8     1         3          0     NA     NA        NA
 9     1         1          4      1    709       725
10     1         2          0      1    698       692
# ℹ 6,315 more rows

• How are we supposed to tell what these values mean?!
• Recode to make the dataset self-documenting

Variables to Recode

Recoding Variables

• Use case_match() from dplyr to recode values

star <- star |> 
  mutate(classtype = case_match(classtype,
                                1 ~ 'small',
                                2 ~ 'regular',
                                3 ~ 'regular+aid'))
star

# A tibble: 6,325 × 6
    race classtype   yearssmall hsgrad g4math g4reading
   <dbl> <chr>            <dbl>  <dbl>  <dbl>     <dbl>
 1     1 regular+aid          0     NA     NA        NA
 2     2 regular+aid          0     NA    706       661
 3     1 regular+aid          0      1    711       750
 4     2 small                4     NA    672       659
 5     1 regular              0     NA     NA        NA
 6     1 regular+aid          0     NA     NA        NA
 7     1 small                4     NA    668       657
 8     1 regular+aid          0     NA     NA        NA
 9     1 small                4      1    709       725
10     1 regular              0      1    698       692
# ℹ 6,315 more rows

💪 Exercise D - (3 min)

Recode the race and hsgrad variables from star as indicated above.

Methods for Producing a Table

Ranked from worst to best:

1. Take a screenshot of STATA output and stretch it so the text is distorted, grainy, and totally unreadable.
2. Copy-and-paste individual results into Excel table, edit manually, and paste it into a Word document.
3. Automatically export regression results into Excel, manually edit the table, and paste into a Word document.
4. Use reproducible tools like knitr::kable, gt and modelsummary to generate and format tables, and include them in any document format.

Publication Quality Tables

• Today: datasummary and knitr::kable
• Future lectures: modelsummary
• Some prefer stargazer. You can use this if you prefer.
• For the ultimate in precision table creation, try gt.