Exploratory Data Analysis

Viewing missing data graphically

It may be useful to view missing data graphically. This may be useful to see if there are specific trends in the data in relation to the missing values. A few ways to plot these data may be useful.

First, it is always a good rule to explore missing data in relation to other variables in the data. If there is evidence that another variable is influencing whether a value is missing, additional statistical controls are needed to adjust for these concerns.

For example, we could explore if the scheduled arrival time is related to whether the actual arrival flight time is missing.

flights %>%
  mutate(
    miss_arrival = is.na(arr_time)
  ) %>%
  ggplot(mapping = aes(sched_arr_time)) + 
  geom_freqpoly(aes(color = miss_arrival)) + 
  theme_bw()

Notice that the count metric masks much of what is being visualized here as there are many more flights that arrived compared to those with missing times. To adjust this, we simply need to change the y-axis from counts to density.

flights %>%
  mutate(
    miss_arrival = is.na(arr_time)
  ) %>%
  ggplot(mapping = aes(x = sched_arr_time, y = ..density..)) + 
  geom_freqpoly(aes(color = miss_arrival)) +
  theme_bw()

The two curves are now standardized so that the area under each curve equals 1, which in turn makes comparison between the two groups easier.

One other special note, for EDA internally, there is no need to spend much time worrying about formatting of the graphics. However, if this plot above would be included in a report or manuscript, this figure would need additional polish to be included.

Qualitative Variables

Bar graphs (frequency tables) are commonly used to explore variation in qualitative variables. For example, if we wished to explore the number of flights that took off for each month of the year from NYC:

ggplot(flights, aes(factor(month))) + 
  geom_bar() + 
  theme_bw()

One special note about the above code, I used the factor() function so that ggplot specifically added all the values of the variable to the x-axis. By default since the month variable is being treated as an integer (a number), it would not show all the values for month.

These counts could be calculated manually with the use of dplyr using the count function. The count function basically creates a frequency table. More complex tables can be created by passing additional variables to the count function.

flights %>%
  count(month)

## # A tibble: 12 × 2
##    month     n
##    <int> <int>
##  1     1 27004
##  2     2 24951
##  3     3 28834
##  4     4 28330
##  5     5 28796
##  6     6 28243
##  7     7 29425
##  8     8 29327
##  9     9 27574
## 10    10 28889
## 11    11 27268
## 12    12 28135

Exercises

1. Copy the code from the bar graph above, but instead of wrapping the month variable in factor, try it without it. What is different? Extra, using the scale_x_continuous function, can you manually add each of the 12 numeric month values to the plot?
2. Using dplyr, manually calculate the number of flights that took off for every day of every month. In other words, how many flights took off everyday of the year. Which day had the most flights?

Quantitative Variables

Histrograms, frequency polygons, or density curves are three common options to explore variation with quantitative variables. Within the flights data, suppose we were interested in exploring the variation in the distance traveled, this could easily be done with a histrogram.

ggplot(flights, aes(distance)) + 
  geom_histogram() + 
  theme_bw()

We could also use a frequency polygon:

ggplot(flights, aes(distance)) + 
  geom_freqpoly() + 
  theme_bw()

We could also use a density curve:

ggplot(flights, aes(distance)) + 
  geom_density() + 
  theme_bw()

When exploring the variation for a single variable overall, I tend to use histograms. However, when attempting to see if the variation changes across values of a categorical variable, histograms are difficult as the groups likely overlap. These are instances when using the frequency polygon or density curves are useful. Here are examples of both when exploring variation differences by month.

ggplot(flights, aes(distance)) + 
  geom_freqpoly(aes(color = factor(month))) + 
  theme_bw()

ggplot(flights, aes(distance)) + 
  geom_density(aes(color = factor(month))) + 
  theme_bw()

You can also calculate the counts plotted in the histograms and frequency polygons using the count as with qualitative variables. We just now need to use the cut_width function to specify bins.

flights %>%
  count(cut_width(distance, 100))

## # A tibble: 28 × 2
##    `cut_width(distance, 100)`     n
##    <fct>                      <int>
##  1 [-50,50]                       1
##  2 (50,150]                    2514
##  3 (150,250]                  36839
##  4 (250,350]                  18442
##  5 (350,450]                  15233
##  6 (450,550]                  29149
##  7 (550,650]                  11688
##  8 (650,750]                  33482
##  9 (750,850]                  19482
## 10 (850,950]                  19644
## # … with 18 more rows

Note, these counts may differ from above as the binwidth was not specifically stated when creating the histrogram or frequency polygon.

It may also be useful to calculate the variance, standard deviation, or the range. These can be calculated using the summarize function.

flights %>%
  summarize(
    var_dist = var(distance, na.rm = TRUE),
    sd_dist = sd(distance, na.rm = TRUE),
    min_dist = min(distance, na.rm = TRUE),
    max_dist = max(distance, na.rm = TRUE)
  )

## # A tibble: 1 × 4
##   var_dist sd_dist min_dist max_dist
##      <dbl>   <dbl>    <dbl>    <dbl>
## 1  537631.    733.       17     4983

You could pair this with the group_by function to calculate these values for different groups (e.g. by month).

Exercises

1. Explore variation in the air_time variable.
2. Does the variation in the air_time variable differ by month?

Two Qualitative Variables

Covariation in two qualitative variables is more difficult to view visually due to the restricted possible values in each variable. Suppose for example, we wished to explore covariation in the origin of the flight and the carrier.

ggplot(flights, aes(origin, carrier)) + 
  geom_count()

This plot is okay, however, I think a more useful plot is to use a tile plot to explore these differences with color.

flights %>%
  count(origin, carrier) %>%
  ggplot(aes(origin, carrier)) + 
  geom_tile(aes(fill = n)) + 
  theme_bw()

Note, that holes mean missing values (i.e. no flights from that airport from that carrier).

Exercises

1. Explore the covariation between the month and day variables. Note, these are treated as continuous in the data, but in reality they are likely best represented as qualitative.

Two Quantitative Variables

Scatterplots are useful for two quantitative variables. Suppose for example that we wish to explore the relationship between the air_time variable and the arr_delay variable. This could be done with a scatterplot.

ggplot(flights, aes(air_time, arr_delay)) + 
  geom_point() + 
  theme_bw()

## Warning: Removed 9430 rows containing missing values (geom_point).

One problem with the plot above, is overplotting. There are two fixes for this, one is to use transparent points using the alpha argument to the geom_point function.

ggplot(flights, aes(air_time, arr_delay)) + 
  geom_point(alpha = .05) + 
  theme_bw()

## Warning: Removed 9430 rows containing missing values (geom_point).

Another approach that will simplify the exploration is to use boxplots. This will involve grouping the air_time variable into “bins.”

ggplot(flights, aes(x = air_time, y = arr_delay)) + 
  geom_boxplot(aes(group = cut_width(air_time, 50))) + 
  theme_bw()

## Warning: Removed 9430 rows containing missing values (stat_boxplot).

Even another more sophisticated graphic is to do the quantitative alternative to geom_tile. Note, the code below uses the hexbin package, but a similar function is geom_bin2d.

# install.packages("hexbin")
library(hexbin)
ggplot(flights, aes(x = air_time, y = arr_delay)) + 
  geom_hex()

## Warning: Removed 9430 rows containing non-finite values (stat_binhex).

Adding a Third Variable

Adding a third variable is often useful, but can be difficult to think about procedurally. The type of plot that is useful depends on the type the third variable is. For example, if the third variable is also quantitative, the visualization is more difficult, however, if the third variable is qualitative, there are two main options. These will be explored in more detail below.

The main two approaches for adding a third variable when it is qualitative is to use a different color/shape for the values of this variable or to facet the plot. Suppose we wished to explore the covariation between the following three variables: air_time, arr_delay, and origin. The two different options are shown below.

ggplot(flights, aes(air_time, arr_delay)) + 
  geom_hex() + 
  facet_grid(. ~ origin) + 
  theme_bw()

## Warning: Removed 9430 rows containing non-finite values (stat_binhex).

ggplot(flights, aes(air_time, arr_delay)) + 
  geom_point(aes(color = origin), alpha = .05) + 
  theme_bw()

## Warning: Removed 9430 rows containing missing values (geom_point).

Plotting three quantitative variables commonly involves binning one one the variables to turn it into an ordinal variable with different levels. For example, see the example with two variables and the boxplot, a similar approach could be used to facet by this third variable. Below is a simple example:

ggplot(flights, aes(x = air_time, y = arr_delay)) + 
  geom_hex() + 
  theme_bw() + 
  facet_wrap(~ cut_width(dep_time, 250))

## Warning: Removed 9430 rows containing non-finite values (stat_binhex).

One Quantitative, One Qualitative

This was actually already discussed in the discussion of variation by exploring differences in variation for different levels of a qualitative variable (see above). If the variation differs by groups, there is then evidence of covariation.

Correlations

It is also useful to calculate and visualize raw correlations. To calculate raw correlations (assuming only quantitative variables), the cor function is useful.

flights %>%
  select(air_time, arr_delay, dep_time) %>%
  cor(use = 'pairwise.complete.obs')

##              air_time   arr_delay    dep_time
## air_time   1.00000000 -0.03529709 -0.01461948
## arr_delay -0.03529709  1.00000000  0.23230573
## dep_time  -0.01461948  0.23230573  1.00000000

To visualize a correlation matrix, the GGally package is useful. Note, the ggpairs function can take some time to run.

# install.package("GGally")
library(GGally)
flights %>%
  select(air_time, arr_delay, dep_time) %>%
  na.omit() %>%
  sample_n(1000) %>%
  ggpairs()

Exercises

1. Explore covariation in the dep_delay and arr_delay variables. What type of relationship, if any, appears to be present?
2. Explore the relationship between dep_delay, arr_delay, and origin. What type of relationship is present. Does the relationship between dep_delay and arr_delay differ by origin?
3. Finally, calculate the correlation matrix for dep_delay, arr_delay, and dep_time.

Measures of Central Tendency

Exploring measures of central tendency or simply common values/repeated of common values can also be important.

ggplot(flights, aes(arr_time)) +
  geom_histogram(binwidth = 50) +
  theme_bw()

## Warning: Removed 8713 rows containing non-finite values (stat_bin).

Measures of central tendency can be directly calculated using the summarise function. For example, exploring central tendency of the arr_delay variable.

flights %>%
  summarise(
    avg_arrdelay = mean(arr_delay, na.rm = TRUE),
    med_arrdelay = median(arr_delay, na.rm = TRUE)
  )

## # A tibble: 1 × 2
##   avg_arrdelay med_arrdelay
##          <dbl>        <dbl>
## 1         6.90           -5

More interesting computations can be performed by using adding in the group_by function.

Exercises

1. Using the txhousing data, explore rare/common cases in the median sale price for the following 3 cities: Austin, Dallas, and Houston.
2. Using the data from #1, explore measures of central tendency in the median sale price of these three cities. How have these changed over time (year)?
3. Create an effective visualization that explores differences in the median sale price over time for these three cities.