Chapter 1 Point Estimation

1.1 Overview

Suppose we have an unknown population parameter, such as a population mean \(\mu\) or a population proportion \(p\), which we’d like to estimate. For example, suppose we are interested in estimating:

  • \(p\): the (unknown) proportion of American college students, 18-24, who have a smartphone
  • \(\mu\): the (unknown) mean number of days it takes Alzheimer’s patients to achieve certain milestones

In either case, we can’t possibly survey the entire population. That is, we can’t survey all American college students between the ages of 18 and 24. Nor can we survey all patients with Alzheimer’s disease. So, of course, we take a random sample from the population and use the resulting data to estimate the value of the population parameter. However, we want the estimate to be “good” in some way.

In this lesson, we’ll learn two methods, namely the method of maximum likelihood and the method of moments, for deriving formulas for “good” point estimates for population parameters. We’ll also learn one way of assessing whether a point estimate is “good.” We’ll do that by defining what it means for an estimate to be unbiased.

1.1.1 Definitions

We’ll start the lesson with some formal definitions. In doing so, recall that we denote the \(n\) random variables arising from a random sample as subscripted uppercase letters: \(X_1, X_2, \ldots, X_n\). The corresponding observed values of a specific random sample are then denoted as subscripted lowercase letters: \(x_1, x_2, \ldots, x_n\).

Parameter Space

The range of possible values of the parameter \(\theta\) is called the parameter space \(\Omega\) (the Greek letter “omega”). For example, if \(\mu\) denotes the mean grade point average of all college students, then the parameter space (assuming a 4-point grading scale) is: \[ \Omega = \{ \mu : 0 \leq \mu \leq 4 \} \] And, if \(p\) denotes the proportion of students who smoke cigarettes, then the parameter space is: \[ \Omega = \{ p : 0 \leq p \leq 1 \} \]

Point Estimator

The function of \(X_1, X_2, \ldots, X_n\), that is, the statistic \(u = (X_1, X_2, \ldots, X_n)\) used to estimate \(\theta\) is called a point estimator of \(\theta\). For example, the function: \[ \bar{X} = \frac{1}{n} \sum_{i=1}^{n} X_i \] is a point estimator of the population mean \(\mu\). The function: \[ \hat{p} = \frac{1}{n} \sum_{i=1}^{n} X_i \] (where \(X_i = 0\) or 1) is a point estimator of the population proportion \(p\). And, the function: \[ S^2 = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})^2 \] is a point estimator of the population variance \(\sigma^2\).

Point Estimate

The function \(u(x_1, x_2, \ldots, x_n)\) computed from a set of data is an observed point estimate of \(\theta\). For example, if \(x_i\) are the observed grade point averages of a sample of 88 students, then: \[ \bar{x} = \frac{1}{88} \sum_{i=1}^{88} x_i = 3.12 \] is a point estimate of \(\mu\), the mean grade point average of all the students in the population. And, if \(x_i = 0\) if a student has no tattoo, and \(x_i = 1\) if a student has a tattoo, then: \[ \hat{p} = 0.11 \] is a point estimate of \(p\), the proportion of all students in the population who have a tattoo.

Now, with the above definitions aside, let’s go learn about the method of maximum likelihood.

1.2 Maximum Likelihood Estimation

Statement of the Problem

Suppose we have a random sample \(X_1, X_2, ..., X_n\) whose assumed probability distribution depends on some unknown parameter \(\theta\). Our primary goal here will be to find a point estimator \(u(X_1, X_2, ..., X_n)\), such that \(u(x_1, x_2, ..., x_n)\) is a “good” point estimate of \(\theta\), where \(x_1, x_2, ..., x_n\) are the observed values of the random sample. For example, if we plan to take a random sample \(X_1, X_2, ..., X_n\) for which the \(X_i\) are assumed to be normally distributed with mean \(\mu\) and variance \(\sigma^2\), then our goal will be to find a good estimate of \(\mu\), say, using the data \(x_1, x_2, ..., x_n\) that we obtained from our specific random sample.

The Basic Idea

It seems reasonable that a good estimate of the unknown parameter \(\theta\) would be the value of \(\theta\) that maximizes the probability, or rather, the likelihood, of getting the data we observed. So, that is, in a nutshell, the idea behind the method of maximum likelihood estimation. But how would we implement the method in practice? Well, suppose we have a random sample \(X_1, X_2, ..., X_n\) for which the probability density (or mass) function of each \(X_i\) is \(f(x_i;\theta)\). Then, the joint probability mass (or density) function of \(X_1, X_2, ..., X_n\), which we’ll (not so arbitrarily) call \(L(\theta)\) is:

\[L(\theta) = P(X_1=x_1, X_2=x_2, ..., X_n=x_n) = f(x_1;\theta) \cdot f(x_2;\theta) \cdot ... \cdot f(x_n;\theta) = \prod_{i=1}^n f(x_i;\theta)\]

In light of the basic idea of maximum likelihood estimation, one reasonable way to proceed is to treat the “likelihood function” \(L(\theta)\) as a function of \(\theta\), and find the value of \(\theta\) that maximizes it.

Example 1-1

Suppose we have a random sample \(X_1, X_2, ..., X_n\) where:

  • \(X_i = 0\) if a randomly selected student does not own a sports car, and
  • \(X_i = 1\) if a randomly selected student does own a sports car.

Assuming that the \(X_i\) are independent Bernoulli random variables with unknown parameter \(p\), find the maximum likelihood estimator of \(p\), the proportion of students who own a sports car.

Answer

If the \(X_i\) are independent Bernoulli random variables with unknown parameter \(p\), then the probability mass function of each \(X_i\) is:

\[f(x_i;\theta) = p^{x_i} \cdot (1-p)^{1-x_i}\]

for \(x_i = 0\) or 1 and \(0 < p < 1\). Therefore, the likelihood function \(L(p)\) is, by definition:

\[L(p) = \prod_{i=1}^n f(x_i;\theta) = p^{x_1} \cdot (1-p)^{1-x_1} \cdot p^{x_2} \cdot (1-p)^{1-x_2} \cdot ... \cdot p^{x_n} \cdot (1-p)^{1-x_n}\]

for \(0 < p < 1\). Simplifying, by summing up the exponents, we get:

\[L(p) = p^{\sum x_i} \cdot (1-p)^{n - \sum x_i}\]

Now, in order to implement the method of maximum likelihood, we need to find the \(p\) that maximizes the likelihood \(L(p)\). We need to put on our calculus hats now since, in order to maximize the function, we are going to need to differentiate the likelihood function with respect to \(p\). In doing so, we’ll use a “trick” that often makes the differentiation a bit easier. Note that the natural logarithm is an increasing function of \(x\):

Alt Text

That is, if \(x_1 < x_2\), then \(f(x_1) < f(x_2)\). That means that the value of \(p\) that maximizes the natural logarithm of the likelihood function \(\ln L(p)\) is also the value of \(p\) that maximizes the likelihood function \(L(p)\). So, the “trick” is to take the derivative of \(\ln L(p)\) (with respect to \(p\)) rather than taking the derivative of \(L(p)\). Again, doing so often makes the differentiation much easier. (By the way, throughout the remainder of this course, I will use either \(\ln L(p)\) or \(\log L(p)\) to denote the natural logarithm of the likelihood function.)

In this case, the natural logarithm of the likelihood function is:

\[ \log L(p) = \left( \sum_{i=1}^{n} x_i \right) \log(p) + \left( n - \sum_{i=1}^{n} x_i \right) \log(1-p) \]

Now, taking the derivative of the log-likelihood, and setting it to \(0\), we get:

\[ \frac{\partial \log L(p)}{\partial p} = \frac{\sum_{i=1}^{n} x_i}{p} - \frac{n - \sum_{i=1}^{n} x_i}{1-p} \]

Now, multiplying through by \(p(1-p)\), we get:

\[ \left( \sum_{i=1}^{n} x_i \right) (1-p) - (n - \sum_{i=1}^{n} x_i) p = 0 \]

Upon distribution, we see that two of the resulting terms cancel each other out:

\[ \sum_{i=1}^{n} x_i - p \sum_{i=1}^{n} x_i - np + p \sum_{i=1}^{n} x_i = 0 \]

leaving us with:

\[ \sum_{i=1}^{n} x_i - np = 0 \]

Now, all we have to do is solve for \(p\). In doing so, you’ll want to make sure that you always put a hat (“^”) on the parameter, in this case, \(p\), to indicate it is an estimate:

\[ \hat{p} = \frac{\sum_{i=1}^{n} x_i}{n} \]

or, alternatively, an estimator:

\[ \hat{p} = \frac{\sum_{i=1}^{n} X_i}{n} \]

Oh, and we should technically verify that we indeed did obtain a maximum. We can do that by verifying that the second derivative of the log-likelihood with respect to \(p\) is negative. It is, but you might want to do the work to convince yourself!

Now, with that example behind us, let us take a look at formal definitions of the terms:

  • Likelihood Function
  • Maximum Likelihood Estimators
  • Maximum Likelihood Estimates

Definition: Let \(X_1, X_2, ..., X_n\) be a random sample from a distribution that depends on one or more unknown parameters \(\theta_1, \theta_2, ..., \theta_m\) with probability density (or mass) function \(f(x_i; \theta_1, \theta_2, ..., \theta_m)\). Suppose that \(\theta_1, \theta_2, ..., \theta_m\) is restricted to a given parameter space \(\Omega\). Then:

  • When regarded as a function of \(\theta_1, \theta_2, ..., \theta_m\), the joint probability density (or mass) function of \(X_1, X_2, ..., X_n\):

\[ L(\theta_1, \theta_2, ..., \theta_m) = \prod_{i=1}^{n} f(x_i; \theta_1, \theta_2, ..., \theta_m) \]

\((\theta_1, \theta_2, ..., \theta_m)\) in \(\Omega\) is called the likelihood function.

  • If \([u_1(x_1, x_2, ..., x_n), u_2(x_1, x_2, ..., x_n), ..., u_m(x_1, x_2, ..., x_n)]\) is the m-tuple that maximizes the likelihood function, then:

\[ \hat{\theta}_i = u_i(X_1, X_2, ..., X_n) \]

is the maximum likelihood estimator of \(\theta_i\), for \(i = 1,2,...,m\).

  • The corresponding observed values of the statistics in (2), namely:

\[ [u_1(x_1, x_2, ..., x_n), u_2(x_1, x_2, ..., x_n), ..., u_m(x_1, x_2, ..., x_n)] \]

are called the maximum likelihood estimates of \(\theta_i\), for \(i = 1,2,...,m\).

Example 1-2: Suppose the weights of randomly selected American female college students are normally distributed with unknown mean \(\mu\) and standard deviation \(\sigma\). A random sample of 10 American female college students yielded the following weights (in pounds): 115, 122, 130, 127, 149, 160, 152, 138, 149, 180.

Based on the definitions given above, identify the likelihood function and the maximum likelihood estimator of \(\mu\), the mean weight of all American female college students. Using the given sample, find a maximum likelihood estimate of \(\mu\) as well.

Answer: The probability density function of \(X_i\) is:

\[ f(x_i; \mu, \sigma^2) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left[-\frac{(x_i - \mu)^2}{2\sigma^2}\right] \]

for \(-\infty < x < \infty\). The parameter space is \(\Omega = \{ (\mu, \sigma) : -\infty < \mu < \infty \text{ and } 0 < \sigma < \infty \}\). Therefore, the likelihood function is:

\[ L(\mu, \sigma) = \sigma^{-n} (2\pi)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i - \mu)^2\right] \]

for \(-\infty < \mu < \infty\) and \(0 < \sigma < \infty\). It can be shown, upon maximizing the likelihood function with respect to \(\mu\), that the maximum likelihood estimator of \(\mu\) is:

\[ \hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} X_i = \bar{X} \]

Based on the given sample, a maximum likelihood estimate of \(\mu\) is:

\[ \hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i = \frac{1}{10} (115 + \ldots + 180) = 142.2 \]

pounds. Note that the only difference between the formulas for the maximum likelihood estimator and the maximum likelihood estimate is that:

  • the estimator is defined using capital letters (to denote that its value is random), and
  • the estimate is defined using lowercase letters (to denote that its value is fixed and based on an obtained sample).

Okay, so now we have the formal definitions out of the way. The first example on this page involved a joint probability mass function that depends on only one parameter, namely P, the proportion of successes. Now, let’s take a look at an example that involves a joint probability density function that depends on two parameters.

Example 1-3: Let \(X_1, X_2, ..., X_n\) be a random sample from a normal distribution with unknown mean \(\mu\) and variance \(\sigma^2\). Find maximum likelihood estimators of mean \(\mu\) and variance \(\sigma^2\).

Answer: In finding the estimators, the first thing we’ll do is write the probability density function as a function of \(\theta_1 = \mu\) and \(\theta_2 = \sigma^2\):

\[ f(x_i; \theta_1, \theta_2) = \frac{1}{\sqrt{\theta_2} \sqrt{2\pi}} \exp\left[-\frac{(x_i - \theta_1)^2}{2\theta_2}\right] \]

for \(-\infty < \theta_1 < \infty\) and \(0 < \theta_2 < \infty\). We do this so as not to cause confusion when taking the derivative of the likelihood with respect to \(\sigma^2\). Now, that makes the likelihood function:

\[ L(\theta_1, \theta_2) = \prod_{i=1}^{n} f(x_i; \theta_1, \theta_2) = \theta_2^{-\frac{n}{2}} (2\pi)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\theta_2} \sum_{i=1}^{n} (x_i - \theta_1)^2 \right] \]

and therefore the log of the likelihood function:

\[ \log L(\theta_1, \theta_2) = -\frac{n}{2} \log \theta_2 - \frac{n}{2} \log(2\pi) - \frac{\sum{(x_i - \theta_1)^2}}{2\theta_2} \]

Now, upon taking the partial derivative of the log likelihood with respect to \(\theta_1\), and setting to 0, we see that a few things cancel each other out, leaving us with:

\[ \frac{\partial \log L(\theta_1, \theta_2)}{\partial \theta_1} = \frac{-2\sum{(x_i - \theta_1)}(-1)}{2\theta_2} \stackrel{\text{set}}{=} 0 \]

Now, multiplying through by \(\theta_2\), and distributing the summation, we get:

\[ \sum{x_i - n\theta_1} = 0 \]

Now, solving for \(\theta_1\), and putting on its hat, we have shown that the maximum likelihood estimate of \(\theta_1\) is:

\[ \hat{\theta}_1 = \hat{\mu} = \frac{\sum{x_i}}{n} = \bar{x} \]

Now for \(\theta_2\). Taking the partial derivative of the log likelihood with respect to \(\theta_2\), and setting to 0, we get:

\[ \frac{\partial \log L(\theta_1, \theta_2)}{\partial \theta_2} = \frac{-n}{2\theta_2} + \frac{\sum{(x_i - \theta_1)^2}}{2\theta_2^2} \stackrel{\text{set}}{=} 0 \]

Multiplying through by \(2\theta_2^2\), we get:

\[ -n\theta_2 + \sum{(x_i - \theta_1)^2} = 0 \]

And, solving for \(\theta_2\), and putting on its hat, we have shown that the maximum likelihood estimate of \(\theta_2\) is:

\[ \hat{\theta}_2 = \hat{\sigma}^2 = \frac{\sum{(x_i - \theta_1)^2}}{n} \]

(I’ll again leave it to you to verify, in each case, that the second partial derivative of the log likelihood is negative, and therefore that we did indeed find maxima.)

In summary, we have shown that the maximum likelihood estimators of \(\mu\) and variance \(\sigma^2\) for the normal model are:

\[ \hat{\mu} = \left(\sum{X_i}\right)/n = \bar{X} \]

and

\[ \hat{\sigma}^2 = \frac{\sum{(X_i - \bar{X})^2}}{n} \]

respectively.

Note that the maximum likelihood estimator of \(\sigma^2\) for the normal model is not the sample variance \(S^2\). They are, in fact, competing estimators. So how do we know which estimator we should use for \(\sigma^2\)? Well, one way is to choose the estimator that is “unbiased.” Let’s go learn about unbiased estimators.