First — the basics:
The term “survey sample” refers to the group of respondents that you desire to take your survey.
Three considerations related to survey sampling are often overlooked:
- Feasibility
- Can you obtain the number of completes needed to achieve your analysis goals?
- Can you obtain the completes in a reasonable amount of field time?
- Is the cost per interview (CPI) within budget?
- Sample Quality
- Sample Make-up
This concept is best described through three questions:
Rule of thumb for basic surveys, using descriptive analytics, is a sample size of n = 100 – 150 for each sub-group that you want to analyze discretely. However, when using advanced statistical analyses, the “n” requirements of your sample will often increase to many more than that. Each analysis technique requires a different sample size to achieve statistical significance. Can you access the number of completes to make your quant. results sound?
This can become a factor where the opinions and experiences of the sample could change, making the respondents who completed the survey first materially different from those who complete it later. For subjects which are stable, say paper products, longer field times are fine. For other subjects, like high profile public figures, opinions literally could change overnight.
This is the payment or compensation a respondent will receive for completing a survey, plus any management costs associated with securing the survey sample. Generally speaking, the harder it is to find and/or access a respondent, the more expensive their CPI fee will be. Costs for survey sample can range from zero to a couple hundred dollars per interview, dependent on the respondent criteria you've defined for your survey. When you scale a CPI fee to 100+ people, sample costs can get steep quick.
Are the people that you desire to take your survey actually taking your survey? The criticality of this sample quality question takes center stage when using outside sample sources.
Fraudulent respondents are real. Therefore, you must take care to build a number of “mouse traps” into your survey to catch them. This means designing screening questions into your survey to determine if the person you are interviewing possesses the characteristics you need to have valid data from the survey. When creating this screener, you want to avoid yes/no or leading questions. You want to ask questions in such a way that only a legitimate respondent should be able to answer the questions correctly. And, where appropriate, you want to add in “foils.”
What do I mean by “foil”? Let’s say you seek to connect with cosmetic users. You might throw a fake brand into a list of brands when asking a potential respondent, “Which of the following brands of cosmetics have you used in the past 6 months?” If the respondent selects the fake brand, they will be disqualified from the survey.
This refers to who is represented in your survey sample. When creating your sample plan, you are making choices on things like quotas, complete minimums or maximums, or mixes. A study’s analysis plan will lead the definition of a study’s sample make-up.
For example, let’s assume a pet supply manufacturer wants to learn more about the housing preferences that pet owners have for three different types of pets — dogs, cats, and dolphins.
If you want to be able to perform subgroup statistical analysis between the dog owners, cat owners, and dolphin owners, then you will want a minimum of n = 300 respondents in the study (n = 100 dog owners, n = 100 cat owners, and n = 100 dolphin owners).
Yet, it’s important to note that this overall sample make up may not be reflective of pet ownership in the market of study. Let’s assume that 48% of pet owners have a dog, 48% own a cat, and 4% own a dolphin.
If we sampled based on representation, we would have n = 144 cat and dog owners each and only n = 12 dolphin owners, much too small to confidently stat test dolphin owner preferences against those of cat and dog owners.
This nuance around sample make-up becomes a problem when you want to be able to report survey results at a total sample level. If we use the total from the evenly split sample, we may conclude there is an amazing high interest in salt water pools. While I may have a very talented cat (Loki), I do not think he would have much interest in having to put on a wet suite to take a nap.
While this is a humorous example, this happens often if a client insists on having say evenly split samples on things like sex/gender, age, education level, or other characteristics which do not conform to the profiles of the population being studied.
Bottom line, pulling together a survey that will address your marketing management questions is not a trivial task. It takes some upfront planning and vigilance to ensure that you are structuring your sample make up to smartly achieve — in a feasible and accurate manner — your project goals.
