
## 3. Estimation in the Bernoulli Model

### Basic Theory

#### Preliminaries

Suppose that $$\bs{X} = (X_1, X_2, \ldots, X_n)$$ is a random sample from the Bernoulli distribution with unknown success parameter $$p \in [0, 1]$$. Thus, these are independent random variables taking the values 1 and 0 with probabilities $$p$$ and $$1 - p$$ respectively. Recall that the mean and variance of the Bernoulli distribution are $$\E(X) = p$$ and $$\var(X) = p (1 - p)$$.

Usually, this model arises in one of the following contexts:

1. There is an event of interest in a basic experiment, with unknown probability $$p$$. We replicate the experiment $$n$$ times and define $$X_i = 1$$ if and only if the event occurred on the $$i$$th run.
2. We have a population of objects of several different types; $$p$$ is the unknown proportion of objects of a particular type of interest. We select $$n$$ objects at random from the population and let $$X_i = 1$$ if and only if the $$i$$th object is of the type of interest. When the sampling is with replacement, these variables really do form a random sample from the Bernoulli distribution. When the sampling is without replacement, the variables are dependent, but the Bernoulli model may still be approximately valid if the population size is large compared to the sample size $$n$$. For more on these points, see the discussion of sampling with and without replacement in the chapter on Finite Sampling Models.

In this section, we will construct confidence intervals for $$p$$. A parallel section on Tests in the Bernoulli Model is in the chapter on Hypothesis Testing. Note that the sample mean of our data vector $$\bs{X}$$ $M = \frac{1}{n} \sum_{i=1}^n X_i$ is the sample proportion of objects of the type of interest. By the central limit theorem, the standard score $Z = \frac{M - p}{\sqrt{p (1 - p) / n}}$ has approximately a standard normal distribution and hence is (approximately) a pivot variable for $$p$$. For a given sample size $$n$$, the distribution of $$Z$$ is closest to normal when $$p$$ is near $$\frac{1}{2}$$ and farthest from normal when $$p$$ is near 0 or 1 (extreme). Because the pivot variable is (approximately) normally distributed, the construction of confidence intervals for $$p$$ in this model is similar to the construction of confidence intervals for the distribution mean $$\mu$$ in the normal model. But of course all of the confidence intervals so constructed are approximate.

As usual, for $$r \in (0, 1)$$, let $$z(r)$$ denote the quantile of order $$r$$ for the standard normal distribution. Values of $$z(r)$$ can be obtained from the special distribution calculator, or from most statistical software packages.

#### Basic Confidence Intervals

For every $$\alpha, \, r \in (0, 1)$$, an approximate $$1 - \alpha$$ confidence set for $$p$$ is $\left\{ p \in [0, 1]: M - z(1 - r \alpha) \sqrt{\frac{p (1 - p)}{n}} \le p \le M - z(\alpha - r \alpha) \sqrt{\frac{p (1 - p)}{n}} \right\}$

Proof:

From our discussion above and the definition of the quantile function, $\P \left[ z(\alpha - r \, \alpha) \le \frac{M - p}{\sqrt{p(1 - p) / n}} \le z(1 - r \, \alpha) \right] \approx 1 - \alpha$ Solving for $$p$$ in the numerator of the fraction in the middle of the inequalities gives the result.

As usual, $$r$$ is the proportion of the significance level $$1 - \alpha$$ in the right tail of the distribution of the pivot variable, and $$1 - r$$ is the proportion of the significance level $$1 - \alpha$$ in the left tail of the distribution of the pivot variable.

The confidence set for $$p$$ is an interval of the form $$\left[U(z(\alpha - r \alpha)), U(z(1 - r \alpha))\right]$$ where $U(z) = \frac{n}{n + z^2} \left(M + \frac{z^2}{2 n} - z \sqrt{\frac{M (1 - M)}{n} + \frac{z^2}{4 n^2}}\right)$

Proof:

This follows by solving the inequalities above for $$p$$. For each inequality, we can isolate the square root term, and then square both sides. This gives quadratic inequalities, which can be solved using the quadratic formula.

The confidence interval is known as the Wilson interval, in honor of Edwin Wilson. As usual, the most important special cases are the equal-tailed $$1 - \alpha$$ confidence interval, obtained by setting $$r = \frac{1}{2}$$, the $$1 - \alpha$$ confidence upper bound, obtained by letting $$r \downarrow 0$$, and the $$1 - \alpha$$ confidence lower bound obtained by setting $$r \uparrow 1$$.

#### Simplified Confidence Intervals

A simplified approximate $$1 - \alpha$$ confidence interval for $$p$$ can be obtained by replacing the distribution mean $$p$$ by the sample mean $$M$$ in the extreme parts of the inequality above

For every $$\alpha, \, r \in (0, 1)$$, an approximate $$1 - \alpha$$ confidence set for $$p$$ is $\left[M - z(1 - r \alpha) \sqrt{\frac{M (1 - M)}{n}}, M - z(\alpha - r \alpha) \sqrt{\frac{M (1 - M)}{n}} \right]$

This confidence interval is known as the Wald interval, in honor of Abraham Wald.. Note that the Wald interval can also be obtained from the Wilson interval by assuming that $$n$$ is large compared to $$z$$, so that $$n \big/ (n + z^2) \approx 1$$, $$z^2 \big/ 2 n \approx 0$$, and $$z^2 \big/ 4 n^2 \approx 0$$.

Bounds

1. An approximate $$1 - \alpha$$ level confidence lower bound for $$p$$ is $M - z(1 - \alpha) \sqrt{\frac{M (1 - M)}{n}}$
2. An approximate $$1 - \alpha$$ level confidence upper bound for $$p$$ is $M + z(1 - \alpha) \sqrt{\frac{M (1 - M)}{n}}$
Proof:

Part (a) follows by letting $$r \uparrow 1$$ and part (b) by letting $$r \downarrow 0$$, in the general two-sided interval above.

Of the general two-sided $$1 - \alpha$$ confidence intervals above, the one with smallest length is the equal-tailed interval obtained by letting $$r = \frac{1}{2}$$: $\left[ M - z\left(1 - \frac{\alpha}{2}\right) \sqrt{\frac{M (1 - M)}{n}}, M + z\left(1 - \frac{\alpha}{2}\right) \sqrt{\frac{M (1 - M)}{n}} \right]$

Proof:

The length of the interval is $[z(1 - r \alpha) - z(\alpha - r \alpha)] \sqrt{\frac{M (1 - M)}{n}}$ So we just have to minimize the first factor. The minimum value occurs at $$r = \frac{1}{2}$$ by properties of the standard normal quantile function.

Note that this interval is symmetric about the sample proportion $$M$$ but that the length of the interval, as well as the center is random. This is the two-sided interval that is normally used.

Use the simulation of the proportion estimation experiment to explore the procedure. Use various values of $$p$$ and various confidence levels, sample sizes, and interval types. For each configuration, run the experiment 1000 times and compare the proportion of successful intervals to the theoretical confidence level.

#### Conservative Confidence Intervals

Note that the function $$m \mapsto m(1 - m)$$ is maximized when $$p = \frac{1}{2}$$ and thus the maximum value is $$\frac{1}{4}$$. We can obtain conservative confidence intervals for $$p$$ from the simplified confidence intervals above by using this fact.

For every $$\alpha, \, r \in (0, 1)$$, a conservative $$1 - \alpha$$ confidence interval for $$p$$ is

$\left[M - z(1 - r \alpha) \frac{1}{2\,\sqrt{n}}, M - z(\alpha - r \alpha) \frac{1}{2\,\sqrt{n}}\right]$

Bounds

1. A conservative $$1 - \alpha$$ level confidence lower bound for $$p$$ is $$M - z(1 - \alpha) \frac{1}{2 \sqrt{n}}$$.
2. A conservative $$1 - \alpha$$ level confidence upper bound for $$p$$ is $$M + z(1 - \alpha) \frac{1}{2 \sqrt{n}}$$.

Of the two-sided $$1 - \alpha$$ conservative confidence intervals in above, the one with smallest length is the equal-tailed interval obtained by letting $$r = \frac{1}{2}$$: $\left[M - z\left(1 - \alpha / 2\right) \frac{1}{2 \sqrt{n}}, M + z\left(1 - \alpha/2\right) \frac{1}{2 \sqrt{n}} \right]$

Proof:

The proof is as before. The length of the interval is $[z(1 - r \alpha) - z(\alpha - r \alpha)] \frac{1}{2 \sqrt{n}}$ The minimum value occurs at $$r = \frac{1}{2}$$ by properties of the standard normal quantile function.

Note that this interval is symmetric about the sample proportion $$M$$ and that the length of the interval is deterministic. This is the conservative two-sided interval that is normally used. Of course, the conservative confidence intervals will be larger than the approximate confidence intervals. The conservative estimate can be used to design the experiment. Recall that the margin of error is the distance between the sample proportion $$M$$ and an endpoint of the confidence interval. We just consider the symmetric, two-sided interval, and the confidence upper and lower bounds.

A conservative estimate of the sample size $$n$$ needed to estimate $$p$$ with confidence $$1 - \alpha$$ and margin of error $$d$$ is given in the equation below, where $$z_\alpha = z(1 - \alpha / 2)$$ for the two-sided interval and $$z_\alpha = z(1 - \alpha)$$ for the confidence upper or lower bound: $n = \left\lceil \frac{z_\alpha^2}{4 d^2} \right\rceil$

Proof:

With confidence level $$1 - \alpha$$, the margin of error is $$z_\alpha \frac{1}{2 \sqrt{n}}$$. Setting this equal to the prescribed value $$d$$ and solving gives the result.

### Computational Exercises

In a poll of 1000 registered voters in a certain district, 427 prefer candidate X. Construct the 95% two-sided confidence interval for the proportion of all registered voters in the district that prefer X.

$$(0.396, 0.458)$$

A coin is tossed 500 times and results in 302 heads. Construct the 95% confidence lower bound for the probability of heads. Do you believe that the coin is fair?

0.579. No, the coin is almost certainly not fair.

A sample of 400 memory chips from a production line are tested, and 30 are defective. Construct the conservative 90% two-sided confidence interval for the proportion of defective chips.

$$(0.034, 0.116)$$

A drug company wants to estimate the proportion of persons who will experience an adverse reaction to a certain new drug. The company wants a two-sided interval with margin of error 0.03 with 95% confidence. How large should the sample be?

1068

An advertising agency wants to construct a 99% confidence lower bound for the proportion of dentists who recommend a certain brand of toothpaste. The margin of error is to be 0.02. How large should the sample be?

The Buffon trial data set gives the results of 104 repetitions of Buffon's needle experiment. Theoretically, the data should correspond to Bernoulli trials with $$p = 2 / \pi$$, but because real students dropped the needle, the true value of $$p$$ is unknown. Construct a 95% confidence interval for $$p$$. Do you believe that $$p$$ is the theoretical value?
$$(0.433, 0.634)$$. The theoretical value is approximately 0.637, which is not in the confidence interval.