The Principle Behind the Data: Understand Maximum Likelihood Estimation with Simple Examples

    

Have you ever wondered how statisticians can make statements about an entire population by studying only a small sample? The secret lies in methods such as Maximum Likelihood Estimation—a powerful technique that helps us make the “best guess” about unknown parameters.

What is Statistical Inference?

Statistical inference means obtaining information from a sample and, based on that, drawing conclusions about characteristics of the entire population from which the sample was taken. Among several methods that produce good estimators, today we will focus on the maximum likelihood method.

An Intuitive Example

Imagine a box with many blue and orange balls. You don't know which color is more frequent, but you know there are only two possibilities:

1.   Three blue balls for every orange ball → probability of blue: p = ¾

 

2.   One blue ball for every three orange balls → probability of blue: p = ¼

 

Now, you draw 3 balls with replacement and observe how many are blue. How do you decide what the true value of p is? See Table 1. 

Table 1. The Probabilities at Play

Nº of blue balls	p = ¾	p = ¼
0	1/64	27/64
1	9/64	27/64
2	27/64	9/64
3	27/64	1/64

Figure 1. The Probabilities at Play

 

The strategy is simple: we choose the value of p that makes our observation most likely.

•    If 0 or 1 blue ball comes out → we estimate p = ¼

•    If 2 or 3 blue balls come out → we estimate p = ¾

You just used the maximum likelihood estimator!

From the Specific Case to the General

In the real world, we rarely have only two options. In an experiment with “success” or “failure” outcomes, we might have a sample like: S, S, F, F, S, F (3 successes in 6 trials).

The intuitive approach would lead us to calculate:

                                                   p̂ = x / n = 3 / 6 = 1/2

This is not only a reasonable choice—it is the maximum likelihood estimate. For n = 6 trials, the value p = ½ makes the observation of x = 3 successes the most likely of all possibilities. See Table 2.

Table 2: Probabilities associated with the occurrence of x successes in samples of size n = 6

P value		Number of success
	0	1	2	3	4	5	6
p = 1/2	0,01563	0,09375	0,23438	0,3125	0,23438	0,09375	0,01563

Figure 2: Probabilities associated with the occurrence of x successes in samples of size n = 6

Why Does This Matter?

The maximum likelihood estimator is:

•  Intuitive: It chooses the parameter that maximizes the chance of observing what we actually observe.

•    Powerful: It can be applied to many statistical models that are much more complex than the simple binomial example.

•   Consistent: With large samples, it tends to converge to the true value of the population parameter.

•    Versatile: It forms the basis for a large number of modern statistical techniques used in data science, machine learning, and scientific research.

Practical Examples of Maximum Likelihood Estimation

Example 1: Screw Factory Quality Control

In a screw factory, quality control is performed by selecting a sample from each batch and checking how many are non-conforming. Consider that in a batch of 500 screws, 38 non-conforming ones were found.

·  What is the maximum likelihood estimator for the proportion of non-conforming screws?

·  What is the estimate obtained for this specific batch?

Solution:
The maximum likelihood estimator (MLE) for a proportion p in a binomial distribution is the sample proportion itself, given by the formula p̂ = x / n, where:

·  x is the number of "successes" (in this context, finding a non-conforming screw).

·  n is the sample size.

For this batch:

·  x = 38 (non-conforming screws)

·  n = 500 (total screws in the sample)

The estimate is therefore:
p̂ = 38 / 500 = 0.076 or 7.6%

Conclusion: The maximum likelihood estimate for the proportion of non-conforming screws in the batch is 7.6%.

---

Example 2: Election Poll (Corrected Version)

Election polls are an attempt to capture voters' intentions at a specific moment by conducting a limited number of interviews. It is, therefore, an effort to measure the whole from a part. Imagine a polling institute conducted a preliminary election poll for mayor in a specific municipality. There were two candidates, which we will call A and B.

500 voters were interviewed, yielding the following results:

·  220 votes would be for candidate A

·  180 votes would be for candidate B

·  The remaining voters were undecided.

a) What is the maximum likelihood estimate for the proportion of undecided voters in the population?
b) What is the maximum likelihood estimate for the proportion of votes for candidate A?

Solution:
The same principle applies. The MLE for a population proportion p is the sample proportion p̂ = x / n.

a) Proportion of Undecided Voters:

·  Number of undecided voters in sample (x): 500 - 220 - 180 = 100

·  Sample size (n): 500

·  Estimate: p̂_undecided = 100 / 500 = 0.20 or 20%

b) Proportion of Votes for Candidate A:

·  Number of votes for A in sample (x): 220

·  Sample size (n): 500

·  Estimate: p̂_A = 220 / 500 = 0.44 or 44%

Conclusion: Based on this sample, the maximum likelihood estimates are a 20% proportion of undecided voters and a 44% vote proportion for candidate A in the broader population.