Minimize Errors in Survey Research

Colleagues in the exhibition department at a local museum met to review the list of traveling exhibitions available for rent in the next three years. After a rousing discussion, they decided to seek input from museum members, finding out what traveling exhibition topics were of interest to them. Staff put together a short checklist of known exhibitions, asking members to identify those they would pay a separate admission fee to come see, and added two open-ended questions. Once complete, the exhibits director gave the questionnaire to the membership department for distribution to members in an upcoming member survey.

As the membership coordinator looked at the questions, she thought, as long as we’re asking for member feedback, let me add several questions on issues of interest to the membership. She printed a draft and left for a meeting. As a manager of education was making copies of a program handout, he noticed the questionnaire in the copier. What a great opportunity to get member input on educational program planning, he thought! So, he also created survey questions and added them to the questionnaire.

Needless to say, when member feedback was collated and reviewed, the exhibits director was not only surprised by the inclusion of information related to membership and education, but disappointed by the lackluster responses to the questionnaire; few meaningful responses were obtained to any of the questions.

In fairness to museum members, who did their best to respond to each question, the lack of useful responses were a result of poor planning, cluttered layout and design, and poorly worded questions – all ‘errors’ in the parlance of survey research.

One member wrote a note to the staff saying, “I asked my husband what he thought you meant. He thought you meant something totally different than what I thought you meant so I decided not to answer that question because I didn’t want to get it wrong.”

People responding to an online or mailed questionnaire want to ‘get it right’. They want to give you helpful information. As the person creating the questions and designing the instrument, the questionnaire, you need to make providing accurate information easy for those answering your questions. You need to design items and instruments that minimize the potential for errors, or problems, regardless of the method of survey research used.

First and foremost, you must articulate a clear purpose for instigating the questionnaire. What do you want to know as a result of having asked the questions? Once you’ve identified the purpose, you must make certain that every single item on your questionnaire relates directly to that purpose. Never include an item because ‘it would be nice to know’ or because ‘we have extra room’.

Second, research shows that more people will respond to your questionnaire (even if it is lengthy) if the layout is clean and uncluttered. Use white space wisely.

Third, you must recognize that there are many opportunities for things to go wrong when conducting survey research. There are things you can and must do to minimize those errors as you design and develop instruments. What are the advantages and disadvantages of using each deployment method (e.g., online or mailed questionnaire, in-person or phone interview)? What method best matches your intended outcomes? You must make sure that every question or item means the same thing to everyone. Who is going to respond to your questionnaire? How will you reach them? How will you handle those items for which you have no response or for which the response is incomplete? What is social desirability and how does it impact your results? You will discover that there is an art and a science to wording items and questions and to defining an order or creating a context of items in a questionnaire.

Nothing is ever as easy or as simple as it seems, and this seems to be especially true in the area of survey research. There are so many things to consider!

Don’t have time to figure it all out? Unsure of what to do next? If you would like to learn more about how NBRI can help you design and develop a research study, contact us now at 800-756-6168.