Basic Statistics Concepts: Explain Like I Am Five (Maybe Fifteen)

2018-01-02

I was entitled to give intuitive explanations to a friend who is trying very hard to study intro Statistics concepts. Statistics can be confusing, even at intro level. It is important to get the basic ideas straight, and almost always helpful to stick to your questions until you are absolutely clear. Here I am posting some of my explanations, hoping to help those who are troubled by similar questions.

Background

Suppose you want know the mean of students' heights for this year's graduating class. The whole graduating class is your population, and it has mean height $\mu$ and standard deviation of height $\sigma$. However you cannot collect data for everybody in the graduating class. Instead you only collect heights data for a sample, which consists of $n$ students. The sample has mean height $\overline{X}$ and standard deviation of height $\sigma (X)$. Each time you collect a sample, you get a different mean and standard deviation. Therefore both $\overline{X}$ and $\sigma (X)$ are Statistical variables having their own mean and standard deviation, while $\mu$ and $\sigma$ are just constants. You want to know the relationship between your sample mean $\overline{X}$ and population mean $\mu$, as well as that of your sample sd (standard deviation) $\sigma (X)$ and population sd $\sigma$.

How do you estimate population mean $\mu$ from sample mean $\overline{X}$?

The short answer is you collect a student sample of some size $n$, take their average height ($\overline{X}$), and assert that you think $\mu =m$. Why can you make such an assertion? Because the sample mean is an unbiased estimator of the population mean, or, simpler, you expect your sample mean to be the same as the population mean, although you realize you may have a bad sample. How so? Suppose student height is a Statistical variable $X$ with mean $\mu$ and sd $\sigma$, you have independent and identically distributed (iid) $X_i\sim X$ with $i=1\cdots n$. That is, for the purpose of this problem, they are independent and their mean are all $\mu$ and their sd's are all $\sigma$. $$\mathbb{E}(\overline{X})=\mathbb{E}(\frac{\sum _{i=1}^n{X}_i}{n})=\frac{1}{n}\cdot (\sum _{i=1}^n\mathbb{E}(X_i))$$ Let's stop here and examine why. The second step just substitute numbers to the first step. However it should be realized that the third step is true because the expectation $\mathbb{E}$ is a linear operator. It means $$\mathbb{E}(aX+bY)=a\cdot \mathbb{E}(X)+b\cdot \mathbb{E}(Y)$$ where $a$ and $b$ are two constants and $X$ and $Y$ are two Statistical variables. Now it should be clear why the thrid step is true. $$\frac{1}{n}\cdot (\sum _{i=1}^n\mathbb{E}(X_i))=\frac{1}{n}\cdot (n\mu )=\mu$$ Voila! We see $\mathbb{E}(\overline{X})=\mu$. That is, again, the sample mean is expected to be the same as the population mean.

What is $\sigma (\overline{X})=\sigma /\sqrt{n}$?

Now you ask yourself, what if I there are 1000 students in this graduating class, but I am only taking 4 students in my sample? Is my estimate still unbiased? Is the above still true? Yes, although you have a greater possibility of having a bad sample, the estimator (the way you make the estimate by taking the sample mean) is still good and the estimate is still unbiased. An unbiased estimator may be bad if the variance of the estimator is too large. Then how can you shrink the variance? The answer is by taking a large sample. Intuitively this makes sense, the larger the sample, the extreme data are more likely to be compensated by the other extreme when you take the mean. Mathematically speaking, $$\text{Var}(\overline{X})=\text{Var}(\frac{\sum _{i=1}^n{X}_i}{n})=\frac{1}{n^2}(\sum _{i=1}^n\text{Var}(X_i))$$ The third step is true since for two independent Statistical variables $X$ and $Y$, $$\text{Var}(aX+bY)=a^2\cdot \text{Var}(X)+b^2\cdot \text{Var}(Y)$$ Notice that our $X_i$'s are all independent and identically distributed. Therefore we further have, $$\frac{1}{n^2}(\sum _{i=1}^n\text{Var}(X_i))=\frac{1}{n^2}\cdot n{\sigma }^2=\frac{{\sigma }^2}{n}$$ Take square root to both sides, $$\sigma (\overline{X})=\frac{\sigma }{\sqrt{n}}$$

Why sample variance divides by $n-1$ instead of $n$?

Let's continue our story. Now you know you need a large sample to get an unbiased estimator for the mean heights with a low variance. Now you are becoming more ambitious, you want to estimate the population sd $\sigma$. Is you sample variance an unbiased estimator of the population variance?

(TBC)

© The Responsible Adult 2017 - 2025