Schulman Urges Pollsters to Make Research More Policy-Relevant

For Release
May 17, 2003

(Nashville TN, May 17, 2003) -- The nation's leading polling researchers were urged today to use their formidable research skills to make a bigger impact on policy making by being more strategy-focused.

Mark A. Schulman, 2002-2003 President of the American Association for Public Opinion Research (AAPOR), warned that researchers may be in danger of being marginalized because they are too focused on methodological issues and "cutting and pasting questionnaires," while not unlocking policy insight and strategy from their data.

Schulman spoke to almost 800 opinion researchers at the 58th annual AAPOR Conference in Nashville, TN. AAPOR is the nation's largest association of polling researchers.

The AAPOR President cautioned against the growing use of samples size and response rates as "the" major measures of a survey's quality. Referring to this as the "tyranny of the easily measurable," Schulman instead urged the researchers to develop a better understanding of the people they survey, the issues they address, and how their research can improve public policy in our complex age.

He commented, "My test of a study's worth is, simply, did the study inform a decision-making process and provide strategic insights into the issues? Were the conclusions strategically actionable?"

He decried what he described as a tendency for researchers to be too focused on technical issues, such as response rates, and on desk research often at the expense of doing meaningful research. He urged researchers to spend less time "on autopilot at their desks cutting and pasting questions from previous surveys" and more time in the field with real people to understand their perspectives and issues first hand.

"We need to shift how we practice our profession and how we train and socialize the next generation of survey researchers to think of ourselves not just as survey technicians, but as problem solvers worthy of a seat at the decision-makers' table," he commented.

Schulman was particularly critical of what he termed an "obsession" with response rates as the measure of survey quality. He said that this exemplifies the "tyranny of the most easily measurable statistic" because response rate reporting is "blind to the much tougher-to-measure issues of nonresponse error, blind to coverage errors, blind to poor question wording, and blind to sometimes poor study design that fails to understand the people and issues under study."

At the most fundamental level, higher response rates do not always produce more representative surveys. "Our concerns should be more focused on non-response error, particularly, how do non-respondents differ from respondents," he said.

Schulman urged the researchers to rebalance their efforts so that they evaluate not only the methodological soundness of their research, but also the contribution the research makes to decision-makers.

Mark Schulman is President of Schulman, Ronca & Bucuvalas, Inc., one of the nation's leading survey research organizations. The firm, headquartered in New York City, specializes in both public policy evaluation and market research. Special areas of interest include financial services, insurance, telecommunications, health care, transportation, utility, and media research.