Course Outline

segmentGetting Started (Don't Skip This Part)

segmentIntroduction to Statistics: A Modeling Approach

segmentPART I: EXPLORING VARIATION

segmentChapter 1  Welcome to Statistics: A Modeling Approach

segmentChapter 2  Understanding Data

segmentChapter 3  Examining Distributions

segmentChapter 4  Explaining Variation

segmentPART II: MODELING VARIATION

segmentChapter 5  A Simple Model

segmentChapter 6  Quantifying Error

segmentChapter 7  Adding an Explanatory Variable to the Model

segmentChapter 8  Models with a Quantitative Explanatory Variable

segmentPART III: EVALUATING MODELS

segmentChapter 9  Distributions of Estimates

segmentChapter 10  Confidence Intervals and Their Uses

10.7 Using Confidence Intervals to Evaluate a Group Difference

segmentChapter 11  Model Comparison with the F Ratio

segmentChapter 12  What You Have Learned

segmentResources
list Introduction to Statistics: A Modeling Approach
Using Confidence Intervals to Evaluate a Group Difference
Up to this point we have used confidence intervals to estimate the range of possible values for the population mean (or, if we are using the empty model, \(b\_0\)). But confidence intervals can also be constructed around other parameters—often parameters that we care more about.
Let’s return to the twogroup model that we developed in Chapter 7. Recall we constructed a variable with two levels, Height2Group, that divided our sample into two groups, Short and Tall. We were investigating whether some of the variation in thumb length might be explained by which height group a person was in.
We fit the Height2Group model using the following R code:
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
Height2Group.model
Here is the model fit that resulted:
L_Ch10_UsingC_1
The twogroup model we fit can be represented like this in the notation of the General Linear Model:
\[Y_{i}=b_{0}+b_{1}X_{i}+e\_{i}\]
Recall that \(X\_i\) would take the value 0 for people categorized as Short, and 1 for those deemed Tall. So, the parameter estimate \(b\_1\) represents the observed magnitude of the difference in mean thumb lengths between the two groups.
Why We Would Want a Confidence Interval
We can construct a confidence interval for our estimate of \(b\_1\) using the same method we used previously to get a confidence interval around the mean. Before we construct that confidence interval, though, it is useful to think about why we would want one.
Although the researchers observed a difference in means between the two groups (Short and Tall), it’s possible that the difference was simply due to sampling variation. If we did our thumb length and height group study with another group of 157 students, we would come up with different estimates for \(b_{0}\) and \(b_{1}\). What we really want to know is, what is the true difference in means (\(\beta\_1\)) between the two groups in the population?
Let’s do a little thought experiment to help us see why this is important. Suppose we have samples from two groups (just as we do here), but suppose we know for a fact that the two populations have identical means. In other words, we know that the true value of \(\beta\_1\) is 0. If we take multiple samples from these two populations, we will get various estimates of the group difference. Importantly, although the estimates would be expected to cluster around 0, we now know for a fact that they would vary; the chance of getting a mean difference of exactly 0 would be relatively small.
By constructing a confidence interval for our estimate of \(b\_1\) we can see whether, despite an observed difference in means, it is possible that the true difference in means is 0. In a twogroup comparison, this is generally what we want to know: is the observed difference between the two groups real, or could it just have occurred as a consequence of sampling variation? And if there is a real difference, how big could the difference be? The confidence interval can answer both of these questions.
L_Ch10_UsingC_2
Constructing a Bootstrapped Sampling Distribution of \(b\_{1}\)
To construct a confidence interval around our parameter estimate \(b\_{1}\), we first need to construct a sampling distribution of the estimate. Once we have that, we can use it to find the upper and lower boundaries for the 95% confidence interval around the parameter estimate.
We know from the lm()
model fit above that our sample estimate \(b\_{1}\) is 4.6, meaning that the difference between Short and Tall people’s thumb lengths (or the increment to get from Short to Tall) is 4.6. An easy way to get just this estimate from R is to use the b1()
function.
b1(Thumb ~ Height2Group, data = Fingers)
Combining the b1()
function with the resample()
function we used earlier in the chapter, we can get a single bootstrapped estimate of \(b_{1}\) like this:
b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
L_Ch10_UsingC_3
Let’s go ahead and generate a sampling distribution of \(b_{1}\)s. Add the function do()
to the code below to generate 10,000 bootstrapped increments. We’ll save these into a data frame called bootSDob1 (we made it up to stand for bootstrapped sampling distribution of \(b_{1}\)s). Write some code to look at the first six lines of bootSDob1.
require(supernova)
require(Lock5Data)
require(mosaic)
require(Lock5withR)
custom_seed(51)
Fingers$Height2Group < ntile(Fingers$Height, 2)
Fingers$Height2Group < factor(Fingers$Height2Group, levels = c(1,2), labels = c("short", "tall"))
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
# edit this code to bootstrap 10000 mean differences
SDob1 < do() * b1()
# write code to look at the first few lines of SDob1
# edit this code to boostrap 10000 random mean differences
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# write code to look at the first few lines of bootSDob1
head(bootSDob1)
test_object("bootSDob1")
test_function("head")
test_error()
success_msg("Great work!")
L_Ch10_UsingC_4
In the following DataCamp window, assume that you have already run code to generate bootSDob1. Make a histogram and get the favstats()
for \(b\_{1}\).
require(ggformula)
require(mosaic)
require(supernova)
require(Lock5Data)
require(Lock5withR)
require(okcupiddata)
Fingers$Height2Group < ntile(Fingers$Height, 2)
custom_seed(51)
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# make a histogram of b1
# get the favstats for b1
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# make a histogram of b1
gf_histogram(~ b1, data = bootSDob1, fill = "seagreen", alpha = 1)
# get the favstats for b1
favstats(~ b1, data = bootSDob1)
test_function("gf_histogram")
test_function("favstats")
test_error()
success_msg("Great job, thanks for sticking with it!")
L_Ch10_Constructing_3
Note that the sampling distribution of difference between two means (the estimate \(b\_{1}\)) is normal in shape, just like the sampling distribution of means. The mean of the sampling distribution is close to 4.6, as it should be given that we are resampling from our sample observations, and the mean difference between Short and Tall in our sample was 4.6. The spread of this sampling distribution will give us an estimate of the standard error of the difference between the two groups.
Constructing the Confidence Interval of \(b\_{1}\)
Just as we did before, we can assume that the sampling distribution of the upper and lower bound \(\beta\_{1}\)s (lighter green in the picture above) will correspond to the upper and lower 2.5% cutoffs of the sampling distribution centered at the sample estimate (pictured in darker green).
The bootstrapped sampling distribution of the estimate is centered at the sample estimate so we can use it to get the values of the confidence interval and calculate the critical distance.
We bootstrapped 10,000 increments so we can arrange them in order to find the cutoff points for the highest and lowest 250 increments. To do this we can arrange the bootstrapped \(b\_{1}\)s in descending order, and then look at the 250th from the top and bottom of the distribution.
In this DataCamp window, arrange b1 in bootSDob1 in order, and then print out the 250th and 9,750th values.
require(supernova)
require(Lock5Data)
require(mosaic)
require(Lock5withR)
custom_seed(51)
Fingers$Height2Group < ntile(Fingers$Height, 2)
Fingers$Height2Group < factor(Fingers$Height2Group, levels = c(1,2), labels = c("short", "tall"))
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# arrange b1s in descending order
bootSDob1 <
# print the 250th b1
# print the 9750th b1
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# arrange b1s in descending order
bootSDob1 < arrange(bootSDob1, desc(b1))
# print the 250th b1
# print the 9750th b1
bootSDob1$b1[250]
bootSDob1$b1[9750]
test_object("bootSDob1")
test_output_contains("bootSDob1$b1[250]")
test_output_contains("bootSDob1$b1[9750]")
test_error()
success_msg("Great work!")
L_Ch10_Constructing_4
The 95% confidence interval for the true population difference in thumb lengths between Short and Tall people is between 1.9 and 7.2 millimeters.
We can also find that the critical distance from each of these cutoffs to the mean of the bootstrapped sampling distribution is a little bigger than 2.6. We can think about this critical distance to represent how far above (and below) the sample increment lies the upper bound (and lower bound).
Interpreting the Confidence Interval of \(\beta\_{1}\)
L_Ch10_InterpretingB_1
Because 0 is not included in the 95% confidence interval, we are quite confident that the true difference between the two groups is unlikely to be 0 (though there is a 5% chance we could be wrong). This being the case, we are probably going to want to maintain the more complex model (that is, the Height2Group model).
If 0 was included in our 95% confidence interval, we could consider it within the bounds of likely that there is no real difference in the population parameter between the Short group and the Tall group. This would have implications for which model we might prefer to represent the relationship between height and thumb length. If, in the twogroup model, \(\beta\_1\) turned out to be 0, then we wouldn’t need the \(\beta_{1}X_{i}\) term in our model (0 times either 0 or 1 would still yield a 0).
As you can see, if the true increment in the population is 0, then we would be left with the empty model, which is a simpler model:
\(Y_{i}=\beta_{0}+\cancel{\beta_{1}X_{i}}+\epsilon\_{i}\), which then becomes:
\[Y\_i=\beta\_0+\epsilon\_i\]
We would probably prefer the simpler model unless we were 95% certain that the group difference was real. If we were pretty confident (say 95% confident) that \(\beta\_{1}\) could not be 0, then we could reject the simple model in favor of the more complex one.
Using a Mathematical Model to Construct the Confidence Interval for \(b\_{1}\)
As with the mean before, there is a way to shortcut all of this. Try the function confint()
on Height2Group.model.
require(supernova)
require(Lock5Data)
require(mosaic)
require(Lock5withR)
custom_seed(51)
Fingers$Height2Group < ntile(Fingers$Height, 2)
Fingers$Height2Group < factor(Fingers$Height2Group, levels = c(1,2), labels = c("short", "tall"))
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# here we fit the Height2Group model of Thumb
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
# try out confint() on this model
bootSDob1 < do(10000) * b1(Thumb ~ Height2Group, data = resample(Fingers, 157))
# here we fit the Height2Group model of Thumb
Height2Group.model < lm(Thumb ~ Height2Group, data = Fingers)
# try out confint() on this model
confint(Height2Group.model)
test_object("Height2Group.model")
test_function_result("confint")
test_error()
success_msg("Keep up the good work!")
Notice that with a more complex model, we now get two sets of confidence intervals.
L_Ch10_UsingM_1
Because it is reasonable to assume that the sampling distribution of differences in means is approximately normal and well modeled by the t distribution, the confint()
function uses it to find the cutoff points for the lower and upper 2.5% tails of the sampling distribution.
Now you can compare the 95% confidence intervals constructed using the two methods: bootstrapping and confint()
. Bootstrapping estimated the confidence interval as between 1.9 and 7.2. Pretty close to what we found from modeling the sampling distribution as a t distribution (1.9, 7.3).
L_Ch10_UsingM_2
Just like when we found the confidence interval through resampling, we would say that the complex model is different from the empty model and we should make different predictions of thumb length for short and tall people.