by Leslie MacKenzie
Newsworthy 1999

There's an old journalism joke: "I'm not good at math. Why do you think I went to J-school?" On this count perhaps journalists resemble the average American. American students continue to lag in math and science.

But news reports often include statistical information and poll results: Do you believe Monica Lewinsky? Should President Clinton be impeached? Should the U.S. invade Iraq? Media outlets often conduct their own polls of readers or viewers. How are we to judge the validity of those polls... and why are they useful?

In a recent on-line discussion on polling, ABC News polling analyst Gary Langer chatted with readers about these questions.

One on-line participant asked: "OK, so President Clinton is experiencing high ratings... 69%. My daughter disapproves, my wife disapproves, most people I know disapprove, and if anyone were to ask me, I disapprove. Nobody I know has been asked whether they approve or not. How are these surveys being conducted? Are only so-called 'targeted' audiences being queried?"

Langer responded: "Our polling is based on a true random sample of Americans across the country. We use a computer to generate random telephone numbers so that when we make a call for an ABC News poll, every residential telephone in the country has the same probability of ringing. Your odds are better of hitting the Powerball. Now, your own experience, by contrast, is not random. People tend to live in neighborhoods and associate with friends and relatives who are like-minded.

"...in the '96 election ... our final poll was 2 points off on Clinton's vote, 2 points off on Dole's and 1 point off on Perot's. We interviewed 750 people and told you within a couple of points for each candidate what 90 million were going to do.

"Sampling works. If you don't believe it, next time you go to the doctor and he wants a blood sample, have him take it all."

Sampling works, but it depends on how you get the sample. How valid are the call-in polls seen so often on TV? What about on-line news polls? When an on-line news coordinator asked for tips on conducting an on-line poll, journalists on the Computer-Aided Research and Reporting listserv gave him an earful:

"Why bother? This sort of polling produces results that are essentially meaningless, since the polling is about as unscientific as polling can get [due to its self-selecting sample]," wrote a television news director in Delaware. "The results of these polls (and call-in polls) is DIS-information or NON-information. Why not just run an interactive forum where people can actually post literate comments on specific issues?"

A news editor wrote: "Anyone who even thinks about using an on-line poll for anything other than pure entertainment value should consider the case of Hank, the Angry Drunken Dwarf, who won People Magazine's on-line poll for most beautiful person this year. As I recall, he outpolled Leonardo DiCaprio by a margin of 250,000 to 17,000 thanks to a campaign by Howard Stern's listeners. And, as I recall, this happened even though People was filtering for repeat votes.

"Any on-line poll is so blatantly unscientific and open to manipulation that it is unworthy of even the slightest mention in a serious news story under any circumstances."

The on-line news coordinator acknowledged the shortcomings but still believed there might be value in on-line polling: "While it is true that it isn't scientific and doesn't offer a fair sample of the community's opinion, I think it offers a taste of what people are feeling. After all, aren't the Letters to the Editor sections filled with people who chose to sit down and express their opinion on one subject or another? An on-line poll provides computer users a way to express themselves in the same way that a Letter to the Editor can represent the person who composed it."

Langer agrees: "The on-line medium is an excellent place for us all to post our opinions and to have discussions. It's only the point at which those non-random opinions are tabulated and presented as representative that the exercise becomes meaningless and even misleading."

When Langer was asked what value polls have, he said: "This really is the voice of the people speaking in a way in which it would otherwise not be heard. In the Lewinsky affair, we've had pundits speculating on how average Americans will respond, and we've seen, to a great extent, much of that speculation confounded by actual public opinion. Polls really take the pundits and the spinmeisters and hold their feet to the fire. That's why they've been so important throughout this story...."

So how do we evaluate the information presented to us in news stories? The News Council has heard several complaints of misuse of polls and surveys. In a 1978 complaint about a poll on student attitudes on abortion the Council urged the news media to include the following eight items in poll stories (the list is very similar to Gunaratne's):

1. Who sponsored the survey?
2. What was the exact wording of the question(s) asked?
3. Who was the population surveyed?
4. What was the sample size and the response rate?
5. What is the sampling error?
6. Were some of the results based on only part of the sample?
7. How and when were the opinions collected?
8. Is that too much technical information?
9. Do readers and viewers really want to know this?

Here are some examples Jack Hart, managing editor of The Oregonian, shared with readers of Editor and Publisher (Aug. 15, 1998) about reporting that did not scrutinize these questions.

Survey sponsorship: Consider the AP story that reported the results of a survey and concluded that Americans who think their health insurance needs are covered during retirement may have to turn to welfare. The source of the poll? An insurance company that provided after-retirement medical insurance.

Wording of questions: Consider a 1968 poll that found Lyndon Johnson leading four Republican candidates (in a statewide political race in New York). The poll questions didn't ask about Nelson Rockefeller, New Yorker's favorite son and the clear winner of the race.

Sampling method: To see how the randomness of the sample can be more important than sample size contrast the 750 random ABC News poll respondents who predicted Clinton's win with the (self-selected) two million readers of Literary Digest who picked Alf Landon for president in 1936.

Lacking necessary background, readers and viewers can be seriously misled by this kind of reporting.