You Get What You Pay For: Do It Yourself vs. Full Service Survey Firm

Whether your business is academic, membership, general business, marketing, not-for-profit, for-profit, social, political, product or service related, you will most likely identify the need to conduct a survey at some point. The question becomes, do you do it yourself or work with a full service survey and market research firm?

The answer is: it depends. It depends on whether you have the capability in house to identify the target populations and the expertise to identify the best methods for gathering information, design the instrument(s), collect the data, analyze the data, display the data in a meaningful way to diverse audiences, and write reports of value to multiple stakeholders.

Budget, certainly, can influence whether to do it yourself, work with an online survey company, or a full service survey research firm. Many online survey companies are inexpensive and convenient. They generally offer several levels of service from free, ten question/100 responses with customer support to their top level service and support with an annual fee, unlimited item/unlimited responses, control of survey design, inclusion of your brand, and reporting.

Full service survey research firms work with you every step of the way to articulate the purpose of your survey, work with you to identify your target audience of respondents, write items based on research to minimize the potential for errors, choose the appropriate format of each item to generate valid, reliable responses, analyze data using the most valid statistical measures, write meaningful reports, and work with key individuals in your organization to interpret the results and apply those results to the organization to bring about desired outcomes. Full service survey research firms also have extensive databases which offer you opportunities to benchmark your data against norms of global top performers in your industry.

Here are eight questions to consider when making your decision.

1. What is the purpose of the survey?

Approaching one survey in a purposeful, comprehensive way is better than conducting multiple surveys to gather bits and pieces of information. Mindfully involve all appropriate stakeholders in the discussion as you carefully craft the purpose of your survey. What is it you need to know as a result of conducting the research? What are the desired outcomes?

Are you interested in gathering ideas for quarterly socials after work or do you need to uncover the real reason for low employee morale or customer satisfaction? It is easy enough to ask several questions, list all responses, and identify those responses given most often.

Conducting a root cause analysis, however, requires expertise found in only the top tier survey research firms.

2. From whom do you need to gather information?

Who should you include in the sample? How will the sample be selected? Are there any geographic restrictions? Can false respondents be avoided? That is, can someone other than the invited individual respond to the survey? Do you need to gather information from everyone in your organization or from all of your customers or will a representative sampling suffice? How many people must complete the survey to give the ‘best’ results? Do you need to worry about statistical significance? How do you know?

3. What method(s) are most appropriate to use in gathering information?

Is an online questionnaire the best way to gather data? Will interviews and focus groups supplement information gathered in online questionnaires? If budgets allow, questionnaires may gather initial, broad-based information which is later discussed, refined, and clarified in one on one interviews.

It is important to consider the advantages and disadvantages of using any one method of survey research. And when using online questionnaires, efficacy depends on options provided by the vendor, so be sure to clarify what your selected survey research firm provides.

Advantages of online questionnaires are that they have the potential to be cost efficient with a quick delivery and response time, providing results immediately. You are able to track those who have responded and those who have declined. Some vendors allow respondents to start the questionnaire and complete it at a later time. The venue appeals to tech savvy audiences and is perceived to be environmentally friendly.

Disadvantages, too, depend on options provided by your chosen vendor; it is important to check out the reliability and track record of your vendor. In this world of information overload, we tend to overlook invitations to participate in a survey or direct such invitations directly into our Spam folders. Privacy issues are often of concern, although credible research firms have sophisticated security systems in place. And, as with all questionnaires, there is no way to clarify questions.

Also, will your survey be mobile-optimized for tablets like iPad and Samsung Galaxy and for mobile devices like iPhone and Android? Is your Information Technology Department up to speed on the latest devices?

4. How do you design the questionnaire and minimize the potential for errors?

Every item on the questionnaire must relate to the intent or purpose of the questionnaire. It is tempting to throw in a few questions or items to ‘see what people think’, but resist that temptation. Include only those items that relate directly to the purpose of the questionnaire.

Writing the questionnaire is a science that should be undertaken by those who have the knowledge and skills necessary to minimize the potential for errors. This will ensure that responses given are valid and reliable – essential in trusting gathered feedback.

To avoid potential errors, words on the questionnaire must mean the same thing to everyone. Questions and items must be carefully ordered so as not to influence responses. A context must be created in which information flows smoothly from one section to another.

Respondents want to please those asking the questions; they want to be socially desirable. Therefore, it is important to take that into consideration when asking about socially undesirable behaviors such as unethical practices at work or sensitive issues related to sexual harassment.

Items asking respondents to recall information must be used sparingly, since our brains most often recall that events happened more recently than they actually occurred; this error could influence subsequent decisions and can be avoided by proper wording and ordering on the questionnaire.

Few organizations have the in-house research expertise required to minimize potential errors when writing a questionnaire, the most critical step in the process. Skimping on this step results in misinformation, partial responses, and missed responses. Careful attention to using just the right word, putting items in a logical order, offering diverse formats, creating a context, and minimizing recall yields meaningful responses. That, alone, is worth working with a full service survey research firm.

Another important consideration in questionnaire design is accessibility for people with disabilities. While your place of business may be unaware of the intricacies of laws governing accessibility, full service survey research firms routinely comply with such laws, including the Technology Related Assistance for Individuals with Disabilities Act, the Americans with Disabilities Act, the Telecommunications Act, and the Rehabilitation Act. Do you have the time to make sense of these laws?

5. Do you have what it takes to gather data from your identified sample?

There are many ways to engage participants in a survey questionnaire, so it is critical that you have defined your intended audience and focus only on them. Survey research firms send out email invitations to participate in online surveys, may use pop-up invitations for certain visitors to your website, and customize invitations with your databases. Do you have the capability to extend the invitation in an inclusive way?

It is important to be credible and direct in your invitation to increase the likelihood that your intended respondent both receives the invitation and is motivated to respond.

The beauty of using online questionnaires is that data collection can virtually take place 24/7/365. You must talk with your survey research firm about their capability in gathering responses from various segments of your sample. Do they have the capability to filter results to show feedback from subgroups of the whole? Clarify the type of encryption used to protect your survey link, contents, and responses during transmission.

Do you want to limit access to your survey with a password? Do you want to make some or all questionnaire items required? Who, in your organization, can answer these questions?

6. How will you analyze the information you have collected?

How on earth will you make sense of this seemingly unrelated information? From a business perspective, you know the questions to be answered; from a research perspective, your survey research firm will know the best way in which to analyze the data to give you those answers.

Which key stakeholders will need a copy of the results? How much do they need to know? The answers to these and other questions help inform what methods of analysis to use. Data analysis requires the expertise of skilled practitioners, including researchers and statisticians to determine what statistical analyses will be most meaningful.

Is it enough to show how many people answered each question and display their response? Frequencies are used to report demographics, including the number of people who were involved in a mentoring program, number of managers vs. front line workers, respondents in various geographic divisions of an organization, or various customer segments. Frequencies are good baseline information, but do not make any associations between pieces of gathered information.

The use of cross tabulations makes your data come to life, identifying relationships and trends. Are employees with or without mentors more satisfied with their job? In which geographic division are employees more engaged? Do those who responded by email differ from those who responded through the website?

To find the real reason, or root cause, for low employee morale or customer satisfaction, survey items must be carefully designed and correlations made between each item, followed by complex regression analyses to identify factors causing the problem. While the results are given in conversational language, the analysis process requires specialized understanding of psychological research and higher order statistics.

Full service survey research firms also have the capability to provide benchmarks from their enormous databases against which clients can compare their scores with others in their industry. Such firms also have the resources necessary for your organization to benchmark itself against the best in class in the world, not just within your industry. What a unique way to differentiate yourself!

7. What is the best way to display the results?

As the saying goes, a picture is worth 1,000 words. Using a bar chart, table, line graph, histogram, pie chart, or a scatter plot can allow stakeholders to view results more simply than pages of words. It is important to know who your audience is to best gear the display of results to their level of understanding, making the results relevant to them.

Not all survey research firms offer the same types of data displays or use of your brand and customized colors, designs, and fonts. Do you need the ability to download into other applications like Excel or Access? Will you have the ability to download open-ended question responses? Will you be able to modify data? Will you be able to attach charts to emails and insert graphs or tables into presentations or other documents?

8. What reports have to be written and to whom?

Clearly defining the purpose of the questionnaire will help identify the report recipient(s). Full service firms offer the most support in generating comprehensive reports, differentiating what is needed at various levels of the organization. Some firms also provide training to key members of the organization and other identified stakeholders.

The training not only informs key employees and stakeholders of strategic results, it enlists their help as change agents to move the information throughout the organization. As a result, the survey results and subsequent report become a ‘living being’, stimulating conversation and change within the organization.

A final advantage of using a full service survey research firm is that when you decide to conduct another survey (and you will), the survey research firm can strategically design the second survey to complement the first, capitalizing on analyses to maximize your return on investment.

If you would like to learn more about how the full service survey research services at NBRI can benchmark your services and products to differentiate your organization, contact us now at 800-756-6168.

Terrie Nolinske, Ph.D.

Research Associate

National Business Research Institute