I grumped Wednesday about the new Reason-Rupe poll. The poll asked the usual election questions, but its main focus was opinions the Affordable Care Act. Some of the responses were predictable, some disappointing, and some downright disturbing. But I was the sample demographics that bothered me most:
In other words, the poll grossly underrepresented families earning less than $50,000 (39% in the sample vs. 50% in the population). The pollster reports the sample was “weighted by gender, age, race, Hispanic ethnicity, party identification, Census region, and based on the most recent US Census data.” But not by household income.
Maybe the voices of median- and lower-income Americans just don’t matter enough.
As it happens, the poll included contact information for Reason.org, so I asked Ms. Crissie to email them. I thought they’d be more likely to respond to a human than to a squirrel, even if I am BPI’s roving reporter. And they did respond:
Chris Mitchell, Communications Director here at Reason forwarded me your email. First I wanted to say thank you for your email and thoughtful comments. Chances are the questions you have are likely shared by others, so I’m glad to have an opportunity to address them.
An important point to keep in mind is that social desirability can sometimes impact survey responses. For some survey respondents, questions about income may be considered a sensitive issue. For instance, some may refuse to answer the question, some may inflate their reported household income while others may deflate it because they may be embarrassed to share, are in the company of others, or feel flustered. Although each poll is different, polls often do not weight by income for this reason. For instance, Gallup does not weight by income. Also AAPOR, the American Association for Public Opinion Research, does not include weighting by income as part of its standards.
To your point of ensuring to include groups commonly underrepresented in polls, we weight by gender, age, race, Hispanic ethnicity, party identification, Census region, but also current telephone usage patterns from the National Health Interview Survey, as recommended by AAPOR. In the poll we specifically ask about telephone usage, and among those who use both landline and cell phones, which do they use primarily. We are sure to weight the cell-only group to about 28% as is recommended by AAPOR and found by the National Health Interview Survey. The AAPOR Cell Phone Task Force 2010 writes, “the cell phone only population also includes more renters, a higher proportion of non-whites, and has a lower income as compared to the entire US landline population).”
I hope this answered your questions, and if you have any additional questions, please don’t hesitate to ask.
Emily Ekins, Polling Director, Reason Foundation
I decided to look at the AAPOR website, and it’s true that they don’t mention household income in their guidelines on sample weighting:
Finally, after all of the data have been collected, some simple frequencies are run on certain demographic variables so they can be compared to known characteristics of the population obtained from an external source, such as the U.S. Census. This adjustment is made because some demographic groups tend to be overrepresented or underrepresented in the sample. For example, young men are considerably harder to reach at home than older women. So unweighted data frequently include a larger proportion of older women and a smaller proportion of younger men than what the U.S. Census reports. A pollster typically makes small adjustments – called post-stratification weights – to bring the sample into line with known population characteristics such as age, gender, region and education.
They go into more detail in their online course, but that is a membership-limited website, but I’ll take Ms. Ekins’ word that household income is not one of AAPOR’s weighting standards. I also understand that respondents may not be entirely open or honest about their household incomes. And I’m pleased that Reason-Rupe follow AAPOR guidelines in weighting other factors to ensure the voices of commonly underrepresented groups are heard.
Yet I’m still grumpy that Americans with household incomes less than $50,000 – 50% of the population – comprised only 39% of the polling sample. Perhaps there were a few more. Some may have inflated their incomes, or did not answer the household income question. But we can’t know that for sure. Median- and lower-income Americans are the people who can least afford health care, and for whom the ACA was most specifically tailored. In the Reason-Rupe poll, they received only four-fifths of their rightful voice.
I don’t know whether or how the poll results would have changed had Reason-Rupe weighted for household income. But when academic research shows that “elected officials respond to the views of affluent constituents but ignore the views of poor people,” underrepresenting median- and lower-income families in polls compounds many Americans’ legitimate sense that their voices are not heard in the halls of power.
I hope AAPOR, Reason-Rupe, and other pollsters will consider that legitimate frustration, and begin weighting for household income in their polling samples. “We the People” means all of us.
Good day and good nuts.