## Course Outline

• segmentGetting Started (Don't Skip This Part)
• segmentIntroduction to Statistics: A Modeling Approach
• segmentPART I: EXPLORING VARIATION
• segmentChapter 1 - Welcome to Statistics: A Modeling Approach
• segmentChapter 2 - Understanding Data
• segmentChapter 3 - Examining Distributions
• segmentChapter 4 - Explaining Variation
• segmentPART II: MODELING VARIATION
• segmentChapter 5 - A Simple Model
• segmentChapter 6 - Quantifying Error
• segmentChapter 7 - Adding an Explanatory Variable to the Model
• segmentChapter 8 - Models with a Quantitative Explanatory Variable
• segmentPART III: EVALUATING MODELS
• segmentChapter 9 - Distributions of Estimates
• segmentChapter 10 - Confidence Intervals and Their Uses
• segmentChapter 11 - Model Comparison with the F Ratio
• segmentChapter 12 - What You Have Learned
• 12.2 What You Have Learned About Evaluating Models
• segmentResources

## What You have Learned about Evaluating Models

Wow, look at what you have accomplished! You can come up with questions and hypotheses, fit models, and use fancy statistics like PRE and F in order to quantify how well a model does at explaining variation. But wait—you can do even more!

Our model is the best-fitting model for our data. But what we really want to know is this: Is there really a relationship between income and hate crimes in the Data Generating Process? After all, in any given year, month, or even week, the rate of hate crimes will change and incomes are constantly changing as well. Is this relationship true in the DGP? Or was the relationship in our sample possibly just caused by random chance?

Let’s consider our relatively better model, the income model of hate crimes. Is it really better than the empty model? Is our PRE of .10 significantly higher than we would expect if there were no relationship between income and hate crimes in the DGP?

You now know how to evaluate the income model compared to the empty model. More specifically, you know how to ask whether the relationship we observed between income and hate crimes is truly present in the DGP, or whether it could simply have come about by chance, a result of random sampling variation. It turns out that everything you need to know is right in the supernova table!

L_Ch11_What_8

You now know how to imagine, and instantiate in R, a world where there is no relationship between income and hate crimes! You understand the concept of sampling distribution, and you know something about what supernova() was doing in order to calculate a p of .02.

But you learned how to do much more than just read and interpret the p value in the supernova table. You also learned how to use different techniques for creating sampling distributions—techniques such as bootstrapping, randomization, and simulation.

Let’s use randomization to replicate what we found in the supernova table. To start, let’s create a sampling distribution with 10,000 randomized PREs. We’ve written code in the DataCamp window below to fit the linear model to our sample, and save the sample PRE as samplePRE.

Add code to get 10,000 randomized samples (using shuffle()), and save their PREs in a data frame. Then plot the sampling distribution in a histogram and fill the PREs greater than our sample PRE in a different color. Finally, run a tally() to find the likelihood of getting a PRE greater than our sample statistic just by chance (it might be helpful to get the format in “proportions”).

 #load packages require(ggformula) require(mosaic) require(supernova) require(Lock5Data) require(Lock5withR) require(okcupiddata) require(fivethirtyeight) custom_seed(41)   # we've saved the sample PRE samplePRE <- PRE(avg_hatecrimes_per_100k_fbi ~ median_house_inc, data = hate_crimes) # create sampling distribution of randomized PREs SDoPRE <- # depict the sampling distribution in a histogram # run a tally   # we've saved the sample PRE samplePRE <- PRE(avg_hatecrimes_per_100k_fbi ~ median_house_inc, data = hate_crimes) # create sampling distribution of randomized PREs SDoPRE <- do(10000) * PRE(avg_hatecrimes_per_100k_fbi ~ shuffle(median_house_inc), data = hate_crimes) # depict the sampling distribution in a histogram gf_histogram(~ PRE, data = SDoPRE, fill = ~PRE>samplePRE) # run a tally tally(~PRE > samplePRE, data = SDoPRE, format = "proportion")   test_object("samplePRE") test_function("gf_histogram") test_function("tally") test_error() success_msg("Keep up the hard work") 
DataCamp: ch11-32

You also know how to use randomization to generate a sampling distribution of F ratios, just like you did for PRE. Use the DataCamp window below to create a randomized sampling distribution of F, and then calculate the probability that the F we observed in our sample could have been generated if the empty model is true.

 #load packages require(ggformula) require(mosaic) require(supernova) require(Lock5Data) require(Lock5withR) require(okcupiddata) require(fivethirtyeight) custom_seed(41)   # we've saved the sample F sampleF <- fVal(avg_hatecrimes_per_100k_fbi ~ median_house_inc, data = hate_crimes) # create sampling distribution of randomized Fs SDoF <- # depict the sampling distribution in a histogram # run a tally   # we've saved the sample PRE sampleF <- fVal(avg_hatecrimes_per_100k_fbi ~ median_house_inc, data = hate_crimes) # create sampling distribution of randomized PREs SDoF <- do(10000) * fVal(avg_hatecrimes_per_100k_fbi ~ shuffle(median_house_inc), data = hate_crimes) # depict the sampling distribution in a histogram gf_histogram(~ fVal, data = SDoF, fill = ~fVal>sampleF) # run a tally tally(~fVal > sampleF, data = SDoF, format = "proportion")   test_object("sampleF") test_object("SDoF") test_function("gf_histogram") test_function("tally") test_error() success_msg("Great thinking!") 
DataCamp: ch11-33

Pretty amazing. We started with a mere sample, a brief glimpse of what the Data Generating Process of hate crimes might be. But you showed that there is only a small likelihood (.02) of generating our PRE and F ratio if there is no relationship between income and hate crimes in the world. It’s an amazing thing that you can now take a sample and have sophisticated tools to say something about the population!

But we always want you to stay humble about your statistical know-how. Remember, there are always limits to what you can learn from statistics.

L_Ch11_What_9

## Closing Words

At the beginning of this course we told you that exploring variation, modeling variation, and evaluating models organizes pretty much everything you need to know about statistics. You have truly come a long way since you first heard these vague ideas. You have just demonstrated how much you have grown your statistical skills and understanding! Look at you, asking important questions with data, making amazing plots with just a few keystrokes, fitting models, using fancy GLM notation, creating sampling distributions, and interpreting your analyses with humility. Thank you for all your efforts in grappling with hard concepts. We hope you continue to pursue real understanding in this topic and beyond!