No Fail Recipe for Survey Research

One of our clients enjoys cooking and entertaining. When he has family or friends over for dinner it is not just a meal, it is an event. He wants the occasion to be a very positive and memorable experience for his guests, so he does not do anything half way. Most dishes are made completely from scratch. No one leaves his home hungry.

Those dinners never fail to receive rave reviews. Recently, he was asked where he received his culinary training. His guests were very surprised to learn that had never even taken a cooking class. Everyone wanted to know how he had learned to cook and bake so well without any formal training. Guests were equally surprised by his answer…He simply follows recipes.

With few exceptions, most recipes come out perfect the first time he makes them. And, he has never had a complete disaster in the kitchen while entertaining.

Unfortunately, at NBRI, we are familiar with disasters that have occurred in organizations when attempting to conduct survey research. Today it is no secret that people are the most valuable asset of any business. In order to succeed, we must successfully manage our human resources, both employees and customers. Successful management requires knowledge of the drivers of employee and customer behavior. The only way to obtain this knowledge is through the use of survey research. However, if conducted improperly, survey research can yield inaccurate results; the costs of which multiply as we implement inappropriate interventions. This outcome is easily avoidable. There is a “no fail” recipe for effectively conducting survey research with employees and customers. The recipe has two main ingredients: the survey instrument and the sample.

The Survey Instrument

The results of any survey will only be as good as the questions contained in the survey instrument. You cannot obtain the information you need if you do not ask the proper questions. Sociologist Michael Kimmel, in his book Sociology Now, explains that we must be careful when formulating survey questions because the wording of a question or its location within a survey can change the survey results. Choosing the right questions is a little like selecting ingredients for a recipe. If we choose ingredients that are fresh and of high quality, our recipe is much more likely to be successful. Likewise, if we are careful in how we formulate survey questions and ensure that they are clear and unbiased, survey results are more likely to be accurate and to lead to increased profits. Here are some key things to keep in mind when developing survey questions:

1. The meaning of the question must be clear. When we construct a question, we know what we mean and the type of information we want to obtain. However, our intentions may or may not be clear to the person we are addressing. Have you ever asked someone a question only to receive the following statement in response, “I do not understand what you are asking me.” Take, for example, the following employee survey item “Communication at work is good.” What exactly is meant by “good?” Different employees will interpret this survey item in various ways. To one employee “good” will mean “timely,” to another, “complete,” and to another, “accurate.” The item needs to be more specific to ensure that it is uniformly interpreted and answered correctly.

2. Avoid using “always” and “never” in survey items. For example, “I can always talk to my supervisor about work related problems.” Employees will answer this item in the negative because no one is “always” available to their employees.

3. Ask only one question per survey item. When a survey item asks more than one question, you cannot determine which question the respondent has answered. The item “Training and career planning are available to me” is asking two different questions. It is possible that the employee has plenty of opportunities for training, but few or no opportunities for career planning.

4. Do not imply negative conditions. It is not difficult to put thoughts into peoples’ heads. For example, all I need to do to get a roomful of people to think about pink elephants is to tell them “For the next 30 minutes, do not think about pink elephants. Do not picture a pink elephant or think about the words ‘pink elephant.’” Guess what everyone cannot get out of their thoughts? That’s right, pink elephants. Your survey items can have the same impact. Take for example the survey item “The pace at our company is hectic.” After reading this item, a common reaction is “Now that you mention it, we do have a hectic pace here…it can be exhausting!”

Following these guidelines will enable you to obtain accurate and useful information. But there is another important ingredient: the sample.

Choosing the Right Sample

It is important that your sample accurately represents your population. If a sample is selected haphazardly and is not representative of the population, the results will be useless. According to Floyd Fowler, author of a book titled Survey Research Methods, there are several key factors involved in sample selection:

1. The sample frame. The sample frame is the set of people that has a chance of being selected for your survey, given the sampling approach you are using. From a statistical standpoint, a sample can only be representative of the population that was included in the sample frame. In designing the study, an important issue is how well the sample frame corresponds to the population you want to describe.

2. Probability sampling procedures must be used to designate individual units for inclusion in your sample. Each person must have a known chance of being selected by your sampling procedure. This enables you to determine how well your sample represents your population.

3. The details of sample design. The size and specific procedures for selecting units will influence the precision of your sample estimates. In other words, they affect how closely a sample is likely to approximate the characteristics of the whole population.

A carefully selected sample will accurately represent the characteristics of your population so that you can be confident in taking action based on your survey results.

Analyzing the Data

In addition to having the right “ingredients” when making a recipe, what we do with those ingredients is also important. Let’s say that we are making a glaze for a cake. The main ingredient in the glaze is powdered sugar and the recipe specifies that it be sifted. Well, that is an extra step and requires more time and effort. We will also have one more item, the sifter, to wash when finished. We can choose to skip this step. We can follow all of the other instructions to the letter and use the best ingredients, but if we choose to skip this one step of sifting the powdered sugar, the glaze will have unattractive lumps. Likewise, we can conduct our survey using an excellent survey instrument and a representative sample, but how we analyze the data is a final key step in following our recipe for survey research.

Survey data is typically subjected to descriptive statistical methods. Such methods can be informative in that they can tell us the mean scores on each of our survey items, on topics in the survey, and for the survey as a whole. Unfortunately, if we stop with descriptive methods, we will not be empowered with the information we need to bring about positive changes in our businesses. The knowledge that leads to power comes from inferential statistics. These are the methods that allow us to use our data to predict and manage the behaviors of our population because they identify a few key drivers of employee or customer behavior. Imagine, having to look at the mean scores of dozens of survey items and then to have to decide which items to address first. Inferential statistics take the guesswork out of this process. Armed with the knowledge of several drivers of perceptions, you know exactly where to begin to bring about profitable changes.

On occasion, our client has dinner guests that will ask for one of his recipes. He is always happy to share them, but sometimes after looking at the recipe the person will reply, “That is way too much work. I will never make that.” Many corporate executives feel the same way about survey research. In order to develop, implement, and analyze surveys in-house, you need a team of people who have received education and training in organizational behavior, research methodology, and statistics; preferably at the doctoral level. This is why most businesses choose to outsource their surveys. This brings up another issue – choosing the right research firm to conduct your surveys.

One of the things our client enjoys collecting recipes for is bread. He finds it challenging to make breads of various types and from numerous cultures. In addition to your classic white and wheat breads, he has also learned to make French bread, Italian soda bread, flour tortillas, pitas, and Cuban bread. It amazes him that there is very little variation in the ingredients for these breads. However, the specific amounts of these ingredients and how they are combined, make for very different results. Most research firms have similar ingredients but the outcomes they produce vary considerably. When choosing a firm, here are some key things you should look for:

1. Organizational scientists trained at the doctoral level. If the firm you hire has organizational scientists trained at the doctoral level, they will have the appropriate education and training to design valid and reliable instruments, select representative samples, and accurately analyze data using inferential techniques. They will also understand that it is vital not to cut corners when it comes to survey “ingredients.” For example, some recipes call for “butter or margarine.” Others say “butter, no substitutions.” Failure to heed this warning will most likely result in an inedible product just as failing to follow proper research procedures results in useless data.

2. Standardized survey items/instruments. While some people recommend you develop unique items for your survey, this typically is not necessary. A good research firm has already identified key topics and items for employee and customer surveys and validated their items. Why pay additional money for work that has already been done?

3. Benchmarking data. The value you obtain from your survey results will be exponentially greater if the survey firm has benchmarked it against data from other companies in your industry. This is the only way to derive accurate meaning from your results. Some items traditionally receive lower scores than others. For example, “My pay is appropriate for the work I do.” How many people have you heard complain that they are being paid too much? Benchmarked data informs you what mean scores are typical for each item so that you have a realistic and accurate understanding of each score’s meaning.

Following the “no fail” recipe for survey research will result in a finished product that will be useful and profitable. If you would like to learn how the National Business Research Institute can assist you with this process, contact us today at 800-756-6168.