Margin Of Error

The lesson begins with an introduction to the use of polls in political campaigns. The students are introduced to the term “margin of error”. The students participate in a sampling game in which they try to predict the proportion of chips in a bag that are red. The students learn how margin of error is calculated. The lesson concludes with a reading on questions to ask about a poll.

Any major election in the United States is preceded by extensive campaigns. The campaigns generate media coverage, which usually include polls that try to predict the outcome of the election. Every poll is a sample of the voting population, and because of this, every poll has a margin of error. However, the concept of margin of error is often misrepresented in the media. To properly interpret what a poll is saying, the concept of margin of error must be understood.

POLITICAL CONCEPTS: Polling, Sampling, Margin of Error

• Understand the idea of a poll.
• Estimate the margin of error for a poll using the number of people polled.
• Know what margin of error implies about the results of a poll.
• Be able to list criteria for a good poll.
   

• 4 brown paper bags (Alternative: 9 bags)
• 52 red poker chips (Alternative: 117 red poker chips)
• 28 white poker chips (Alternative: 63 white poker chips)
• 4 calculators (Alternative: 9 calculators)
• Visual 1
   

1. Prepare for the lesson ahead of time by placing 20 poker chips – 13 red and 7 white – in each brown paper bag. The lesson may be done with 4 bags or alternatively 9 bags. The trade-off: More bags mean that students will participate in smaller groups, but results will take a bit longer to tabulate and more chips are required.
    
2. Begin the lesson by asking a series of questions on polls. Ask the students:
   1. If they have recently seen a poll conducted by the media that gave information on an upcoming election. (The students should likely have seen such a poll.)
   2. Why they believe polls are conducted. (Many answers are possible such as: report newsworthy information on a political race, provide the candidates and parties with information, encourage people to become engaged in a political race.)
   3. If they believe the results of a poll. (Answers will vary.) Ask the students who said they do not why they do not believe the polls. (Answers will vary, but will probably focus on the polls being biased by how the poll was conducted or by the organization conducting the poll.)
3. Tell the students that some polls can be biased, and that you will talk to them about that later in the lesson. Tell the students that even polls that are done correctly have what is called a margin of error. Ask the students if they have seen the term margin of error. (The students may have seen the term printed at the bottom of poll results.)
    
4. Tell the students they will conduct an experiment that will help them understand the concept of margin of error.
    
5. Form four (alternatively nine) groups of students. Give each group a bag of chips. Tell the students they should not look in the bag.
    
6. Tell the students the bag contains red and white chips. Tell the students the chips in the bag represent voters for either the red party (a red chip) or the white party (white chip). As pollsters, they are going to try to figure out which party is favored by the voters by polling the voters. Tell them their goal will be to figure out the percentage of voters that favor the red party. (Of course, they will also know then the percentage that favors the white party.) Do not tell them that the bags contain the same percentage of red chips.
    
7. Tell the students to appoint one of their group as the pollster (the one who picks the chip out of the bag) and another as the record keeper. The record keeper will keep a count of the number of red and white chips drawn.
    
8. Tell the students that they will poll voters at random from the bag. Tell them they will do this by shaking the bag (gently), reaching in (without looking) and drawing out one chip. After the chip’s color is noted by the record keeper, the students are to put the chip back in the bag (“with replacement” in statistics jargon).
    
9. Tell the students to draw one chip from their bag and record the color of the chip.
    
10. Ask the students if they can predict the percentage of the chips that are red. (They will likely say that they do not have any information other than if they drew white, it cannot be 100% and if they drew red, it cannot be 0%.) Ask how they could get a better idea of the correct percentage. (They should realize they want to do more sampling.)
     
11. Tell the students to draw 3 more chips as described above, replacing each chip drawn before drawing another chip. Tell them to keep track of the results of all draws. Ask them to predict the percentage of chips that are red based on their results. (The students should, in absence of any other information, report the percentage of red chips they drew from their bag in the four draws – either 0%, 25%, 50%, 75% or 100% in this case.) Ask the students if they have confidence in their estimate. (They should answer that they do not, although they have more information than they did before.)
     
12. Ask the students how they can come up with a better prediction. (They should suggest drawing more chips.)
     
13. Tell the students to draw an additional 12 chips as described above, so that they will have recorded a total of 16 draws from the bag. Give them time to conduct their draws and record their results. Tell them to calculate their estimate of the percentage of chips that are red. (The estimate should be the percentage of red chips drawn.)
     
14. Ask each group to report the number of red chips drawn in the 16 draws, and what percentage of the 16 draws was red. For each group, write two numbers on the board: the total number of red chips drawn and the percentage that were red.
     
15. Tell the students that every poll has a confidence level. The confidence level is the probability that the actual population result will be within the margin of error for the poll. Generally, polls use a 95% level of confidence. This means that on average the margin of error for the poll will cover the true population proportion 95% of the time. In other words, if the poll was conducted 20 times and from each poll result an interval using the margin for error was created, it is likely that in 19 cases the actual population result will be within the interval established using the margin of error. However, polls are not perfect, and it is also likely that one of the twenty poll results will not have the actual population value within its margin of error.
     
16. Tell the students that for a large, random sample of a population, the approximate margin of error for a poll with a 95% level of confidence is by the equation below. Write the equation on the board. Tell the students that n is the number of people surveyed in a poll. Explain that the calculation is 1 divided by the square root of n.

    Margin of error = 1/√n

    (.98/√n is also used. The equation above provides an easily calculated approximation.)
    Note: The number of draws in this lesson (16) is not enough to be considered “large” in a statistical sense. However, 16 draws are enough to demonstrate the principles of the lesson and generate varied results.
     

17. Tell the students they will calculate a confidence interval for their polls using the margin of error. They do this by taking the estimate they found by drawing chips and adding/subtracting the margin of error. Suppose a group drew 8 red chips from their 16 draws. Their estimate of the percentage of chips that are red is 50%. In this case,

    Margin of error = 1/√16 = 1/4 = .25 (= 25%)

    Confidence interval = estimate ± margin of error = 50% ± 25% (or 25% to 75%)
    The margin of error is 25 percentage points. In this example, the poll resulted in 50% red, so the confidence interval for their poll is from 25% to 75%.
     

18. Ask the students to calculate the confidence interval for their poll and report it to you. Write the confidence interval for each group on the board next to their estimates which were reported earlier. The board may look like:

    Group	# Red	% Red	Confidence Interval
    A	10	63%	38% to 88%
    B	10	63%	38% to 88%
    C	11	69%	44% to 94%
    D	12	75%	50% to 100%
    	43

    (Note: If the percentage drawn is higher than 75%, the confidence interval will have an upper end higher than 100%. This is due to the fact that the margin of error formula is simplified for this lesson. If a higher number than 100% is generated, just put 100%, noting that is the highest possible value. Similarly, if the percentage drawn is lower than 25%, then put 0% for the lower end of the confidence interval.)
     

19. Ask the student if the results of their poll are useful. (Again, while the students may suggest they have more information than before, they will view the results as not very useful.)
     
20. Tell the students that the actual percentage of red chips was identical for every bag. Ask the students to look at the results on the board. Based on the results, ask the students to give their estimate of the actual percentage. (Answers will vary, but many students will take an average of the results.)
     
21. Have the students open the bags and find the actual percentage (13 of 20 or 65%). Ask how many of their polls had this value within the confidence interval generated using the margin of error. (Answers will vary.) Note to the students that if a large number of polls were conducted, 95% of the polls, on average, should have the actual population value within their confidence intervals. If one of your polls does not, point out to the students that this is a relatively rare (5%), but not impossible, occurrence.
     
22. Tell the students because they were sampling from the same population (the bags are identical), the results of the polls can be combined into one big poll. Tell the students to calculate the margin of error for this big poll, keeping in mind each group drew 16 chips.

    (Answers:

    For 4 bags, 64 draws: Margin of error = 1/√64 = 1/8 = .125 or about 13%
    For 9 bags, 144 draws: Margin of error = 1/√144 = 1/12 = .083 or about 8% )
     

23. Tell the students to find the percentage of chips drawn that were red for the combined poll. (This is done by adding the number of red chips for each of the (4 or 9) polls and then dividing by the total number of chips drawn (64 or 144). In the example above, four polls are shown. 43 of the 64 draws, or 67%, were red.)
     
24. Ask the students if this information from the combined polls would be useful if we did not know the actual percentage. (The answer may now be yes, since the margin of error is smaller.) Note: In the example above, the drawn percentage of 67% (43 / 64) is close to the actual rate of 65%. The margin of error for the example – 13% – means that the confidence interval is from 54% to 80%. Since a candidate needs 50% to be elected, this might give a candidate confidence that he or she is indeed ahead since the probability of the poll generating a confidence interval that includes the actual value was 95%. Of course, if the percentage draw in your class is lower, the students may note that 50% is within the margin of error, and a politician may be more concerned about whether he or she is ahead.

 

25. Show the students Visual 1. Ask the students why many polls choose to sample 1,000 people instead of 4,000 people. (The cost will be a lot more – perhaps four times as much since four times as many people are being polled – and the margin of error does not shrink a whole lot. The margin of error shrinks by half, but the difference is only 1.6 percentage points.)
     
26. Tell the students that polls may not reflect the actual population because a poll is a sample whose results may be outside the margin of error. It is also possible that a poll’s results may be incorrect because the poll may be incorrectly done.

    Note: The following procedures are based on “20 Questions A Journalist Should Ask About Poll Results,” an article written by the National Council on Public Polls, an association of polling organizations. Its mission is “to set the highest professional standards for public opinion pollsters, and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.” The article is found at http://www.ncpp.org/files/20%20Questions%203rd%20edition_Web%20ver_2006.pdf .
     

27. Tell the students that to be accurate, the poll must randomly survey the population. Ask the students to explain why the following sampling techniques might tend to include some type of people, but leave other people out of the poll.
    1. Survey at upscale shopping malls (oversample the wealthy, leave out poor)
    2. Use a telephone sample (oversample those with phones, leave out those without)
    3. Putting the survey on-line (oversample those with computers – wealthy, young, leave out those without computers, the older and the poorer)
28. Tell the students that it is also important to know who conducted the poll. Polls that are conducted by a partisan group may be intentionally biased. An extreme example may be:

    “How do you prefer for city council, the untrustworthy John Laurence or the wonderful Jane Muggles?”

    While extreme, tell the students this illustrates that how a question is asked matters. In fact, merely rearranging the order of the candidate choices or questions has been shown to influence the results.

Ask the students:

1. Will the actual percentage always be within the margin of error of the poll?

   (No. In the activity in this lesson, it is possible (though unlikely) that the students may draw all red chips – meaning that the actual population proportion is well outside the margin of error. What can be said is that with a 95% confidence level, on average, the actual value will lie within the confidence interval generated using the margin of error for about 95% of the polls conducted.)

2. How many people in a large country need to be randomly polled to get a margin of error of approximately 3%?

   (About 1,000 people.)

3. What are some examples of how polls may be biased, either intentionally or unintentionally?
   (Not conducting a true random sample or biasing the wording of the questions are mentioned in the lesson.)

Readings

A good reference on this topic is:

• Thorton, R. J. and J. A. Thornton. 2004. Erring on the Margin of Error. Southern Economic Journal, vol 71. No. 1., pg. 130-135.

Two references on margin of error and its derivation:

• Becker, William. 1995. Statistics for business and economics. Cincinnati: South-Western/Thompson Learning.
• Becker, William. 1998. Engaging students in quantitative analysis with short case examples from academic and popular press. American Economic Review 88: 480-91.

1. Approximately how many people would need to be surveyed to get a margin of error equal to 10%?
   1. 64
   2. 100
   3. 121
   4. 144
2. You have conducted a poll with a margin of error of 3% and a 95% level of confidence. The results of the poll indicated that 55% of the people polled supported a candidate. Which one of the following is true?
   1. If you conducted the poll repeatedly, 95% of the results will be between 52% and 58%.
   2. The chance of the actual percentage being 55% favoring the candidate is 3%, with 95 % confidence.
   3. The chances of the poll generating a confidence interval containing the actual value was 95% prior to collecting the data.
   4. There is a 95% chance that the actual value is between 52% and 58% and 5% chance that the actual value is outside the 52% and 58% range.
3. Gallup conducts a poll every month over a two year period. The poll has a confidence level of 95% and a margin of error of 3 percentage points. Over the two years, how many times (approximately) will the actual value be outside confidence interval generated using the margin of error of the poll? (One would expect 1 time, since 5% of 24 polls is 1.2.)