Is it Possible to Overcome the Effects of a Poorly Designed Survey Questionnaire?

You’ve put together a questionnaire that is too cluttered and too long. Survey questions were unclear and did not flow. Some survey questions were confusing and did not mean the same thing to everyone. And, respondents were asked to remember dates and time frames that proved difficult for the hardiest of brains. The questionnaire was a mess.

Similarly, we have all tried to make sense of responses to a messy questionnaire. But, if we have garbage in questionnaire design, we get garbage out which makes it impossible to draw conclusions and make business decisions. We could have saved our time and money; sometimes it is a good idea to call on the experts!

How can we overcome a poor survey questionnaire design? It does little good to know that some people loved the food and some hated it without knowing why. We show how many people loved (hated) the food through numeric quantitative methods; the why is usually obtained through inferential analysis that is doomed to failure without good clean data collected by the questionnaire.

Accessibility

Poorly designed questionnaires generally do not take into consideration the needs of people with disabilities or people who are from diverse backgrounds and cultures. People may be, but are not limited to, color blind or may have impaired physical strength, range of motion, sensory, mobility, cognition, vision, speech, or hearing.

Do you and your team have the expertise to provide the questionnaire in alternative formats to be compliant with legislative mandates, including the Americans with Disabilities Act, the Telecommunications Act, and the Rehabilitation Act? If the answer is no, consider using a full service firm who is adept at providing the questionnaire in alternative formats for people from other cultures or for people with disabilities (e.g., foreign language, large print, Braille).

Item Wording and Formatting

Poorly designed questionnaires generally pay little attention to the wording and formatting of items. Example: Rather than ask for a respondent’s age, it is more accurate to ask “What was your age on your last birthday?” This will generate more consistent responses.

Consider the following open-ended item: How much money do you make in a year? First, not everyone has a job. Second, this is a sensitive question.

Are response alternatives mutually exclusive and exhaustive? Consider the response alternatives for the following multiple choice item about salary (i.e., not working, 0-25,000; 25,000-50,000; 50,000-75,000; 75,000-100,000; 100,000 and above). If an individual makes 25,000, will they select the second or third response alternative? Both contain their 25,000 salary! And, ‘not working’ is the same as earning ‘0’ in response alternatives one and two!

A full service survey research firm will work with you to make every word count. The firm might re-word and re-format the salary question offering respondents the following salary ranges from which to choose (i.e., not working, 1-25,000; 25,001-50,000; 50,001-75,000; 75,001-100,000; above 100,000). This wording makes the item more likely to be answered accurately.

Double barreled items are ambiguous and result in inaccurate answers. Consider the following item, “The exhibit was interesting and promoted interaction.” The only people who could accurately respond to that item are those who truly found the exhibit to be interesting and who actually interacted with it. All others would be telling half truths, resulting in misleading results.

The survey firm might re-word the double barreled item into a checklist format: Check all the words you would use to describe the exhibit (interesting, interactive, informative, fun). Separating the double barreled item into two questions would also work.

Do you and your team have the time to identify words that mean the same thing to everyone and format the questionnaire for easy access and response?

Survey Methods

Poorly designed data collection initiatives use less than optimal methods to gather information, resulting in misleading, confusing information.

Online questionnaires are the least expensive way to reach the greatest number of people – globally. Although not everyone has a computer, tablet, or smart phone, computers are available at public libraries and community agencies.

Online questionnaires may look easy to create, but in fact are just as difficult as mailed questionnaires to do well. Email invitations, reminders, and the surveys themselves must be designed to be “responsive” to whatever type of device the respondent uses to take the survey, adjusting text sizes and graphics so that they are easily readable. Once designed, online questionnaires can be easily stored and used from year to year, revising, as necessary.

Obtaining information from in-person interviews may be the most personal approach and most effective way of gaining trust and cooperation from the respondent. It is easy to react to puzzled facial expressions, answer questions, probe for clarification or redirect responses. This type of survey can also be the most expensive to deploy.

Telephone interviews are less expensive than in-person interviews and may be more or less expensive than mailings, depending on numbers involved, yet not everyone has a telephone. And, in this age of identity theft and fraud, many of us are skeptical about sharing information with anyone for any reason over the phone.

A focus group can provide informative feedback as a follow up to other methods or as a method unto itself. Care must be taken, however, to carefully craft questions that are ultimately asked by a trained, unbiased facilitator.

Do you and your team have the expertise to identify and implement the most efficient and effective survey method(s)? Be honest now!

A full service survey research firm works with you to ensure selection of the optimal survey research method to accomplish your goals. Firms often use multiple methods to gather the most reliable, valid information. In an online application, they will also make sure that multiple submissions are not permitted that might skew results.

Sample Selection

Poorly designed survey initiatives select a sample that is either too large or too small. They fail to take into account the diversity of potential respondents. If you ask the wrong people the right questions, you get the wrong answers!

A full service survey research firm will work with you to identify the goals and objectives for using the questionnaire. The research firm will work with you to select the most appropriate respondents for your questionnaire and explain why. They will establish criteria for inclusion and identify the proper method(s) of sample selection to ensure statistically significant and meaningful responses to your questionnaire.

(Non)Response

A poorly designed questionnaire can be the reason for a large number of partial responses to an item or few responses to a large part of the questionnaire.

The most common reason for refusals in survey research used to be reaching people at an inconvenient time on the telephone, largely negated by use of online questionnaires which are available to people 24/7/365. Other common reasons for refusal include lack of time or interest in the subject of the questionnaire.

Do you and your team have the time and expertise to identify non-respondents and identify ways in which they are similar to and different to those who responded to your questionnaire? Careful, there is more to this than meets the eye!

A full service survey research firm will work with you to account for and counter various types of non-response. This ensures that every response is as meaningful and useful as possible in the final data analysis. This includes defining a protocol for handling partially completed items and questionnaires, expanding the sample size, and/or creating an interesting questionnaire. Survey research firms often also obtain information from people who were not asked to complete the questionnaire, to help determine how different the two samples really are. This can give you confidence about how broadly you can generalize your findings. The firm helps you identify what is most relevant to the purpose of your questionnaire.

Context

A poorly designed questionnaire has no context or clearly defined purpose; it is a potpourri of questions thrown together by representatives from every department in the organization.

Do you and your team have the in-house expertise and time to attend to the details of questionnaire design or flow? While not rocket science, it is trickier than it looks!

Working with a full service survey research firm will ensure that your questionnaire has a clear purpose, professional tone, logical order, and uncluttered space.

A poorly designed questionnaire asks questions about sensitive or potentially embarrassing issues while singling out individuals or groups. A poorly designed questionnaire makes people feel that they cannot respond honestly, but need to respond in a way that is ‘socially acceptable’. Therefore, respondents often report smoking less and exercising more, stretching the truth to make themselves look good.

Do you and your team have the expertise to phrase sensitive items so respondents will feel comfortable in responding to them in a straightforward, honest way?

A full service survey research firm works with you to inform respondents about how responses will be used, where and to whom information will be disseminated. Respondents have a right to know whether their answers will be attached to them individually or be reported in group numbers so that people feel comfortable (or not) sharing personal behaviors, opinions, or attitudes.

Item developers at survey firms are expert at crafting items that respondents feel safe answering, rather than sharing what they think people want to hear. When asking about drunken driving, they might rephrase and ask about driving under the influence instead.

A full service firm also has expertise in cross cultural issues, whether aligning word meanings across cultures or accounting for those cultures who are pre-disposed to revealing more information about themselves than others. A survey research firm works with you to guarantee respondent anonymity or confidentiality, which can help equalize socially desirable behaviors and attitudes.

Recall

A poorly designed questionnaire asks respondents to remember what they did weeks, months, and years ago in varied detail. Despite the best of intentions, respondents make errors because they displace events in time, associate the behavior with the wrong time period, or forget entirely. They also might not understand the question.

Information gathered on a questionnaire ultimately depends on the ability to recall experiences or feelings. Do you and your team have the expertise to help respondents recall necessary information without bias or being led to the desired response? This is both an art and a science.

A full service survey research firm is adept at creating items that facilitate recall. Asking people, for example, how often between Memorial Day and Labor Day they go to the beach, defines a time period and helps them frame the question. Longer questions may give people the necessary time to think and produce better responses. Memory can also be stimulated by asking for similar information in different formats throughout the questionnaire. Asking about reactions, attitudes, and behaviors during an actual event also ensures accurate recall.

Reliability / Validity

In poorly designed questionnaires, little to nothing has been done to ensure the reliability and validity of each item and, hence, the entire questionnaire. The questionnaire is not tested prior to its implementation.

Every word in every item on the questionnaire counts. Do you and your team have the time and expertise required to create questionnaires that are both reliable and valid? It is definitely more difficult than it appears.

A full service survey research firm will usually recommend at least one pilot test of the questionnaire to determine whether results yield the same information over time and whether items measure what they were intended to measure.

If you and your team would like to learn more about how the full service survey research services at NBRI can minimize your time and maximize your results, contact us now at 800-756-6168.

Terrie Nolinske, Ph.D.
Research Associate
National Business Research Institute