Posted on

pro-tips-qualitative-sample-sizes

POV: Qualitative Sample Sizes

Purpose

The purpose of this content is to provide a point of view on the ideal sample size for a qualitative market research study, because that’s one of the most common questions even the most seasoned researcher asks us.

Industry norms

Adequate sample sizes have been debated extensively in the market research industry for many years, in part because quantitative sample sizes are statistically easier to measure. Often, quantitative research frameworks get applied to qualitative research because quant norms are concrete (e.g +/- variance), whereas qualitative sample sizes are difficult to prove, mathematically.

A number of studies have been published over the years on the topic of adequate sample sizes for qualitative research. Here are a few:

    • Creswell, Glaser, Morse – all pretty much agree that sample sizes of 30-50 respondents reaches a point of saturation, where adding more respondents does not significantly alter the findings
    • Springer – Springer puts forth the argument that anywhere from 5-50 is adequate, but that 25-30 is considered to be the right number
    • InterQ – recommends 20-30, or even as few as 10

Fabric POV

Qualitative research is more art than science. Bill Bowerman (Nike co-founder) tested his shoe designs on a relatively small sample size, yet was able to scale the innovation to millions of people, in large part because it was the right sample of runners.

Typically, clients turn to qualitative methods to understand meaning – the How and the Why. 

So while we believe there are minimum sample sizes that should be employed in qualitative research, there are some critical variables in the equation we consider:

    • where you are in the process; in our opinion, earlier stage projects (e.g. exploratory product innovation) can employ smaller sample sizes, because more iteration and development will be conducted as the project moves along
    • the business impact: we feel that the higher the business impact of the research, the greater the sampling (qualitative and quantitative combined) should be
    • geography: it can be as easy as domestic and international, but one critical question we consider is how broad the target audience is, geographically
    • research design: how the study is designed can have a huge impact on results (e.g. which questions to ask in what order)
    • research vendor: when you buy a qualitative research resource, you are buying a filter (like a water or air filter). One important variable is how equipped that resource is, down to the people, to field the study and interpret the results
    • quality of recruit: back to the Bowerman example, it’s very important that the people in the study be the right people; beyond technically whether they qualify, we ask ourselves whether the people we interview “feel” like the right consumer. And we will over-index the input of those whom we deem more influential (for example, a starter on an AAU basketball team in Brooklyn may be more insightful than a practice player on a D3 team in the suburbs)
    • analysis: who is interpreting the data is highly important. 
    • methodology: there are a number of qualitative techniques to choose from, including IDIs (individual interviews), focus groups, in-homes, friendship pairs, small group interviews, intercepts, observational research, ethnographies, and digital (online, mobile). In Fabric’s case, the methodology is unique in that it’s effectively “1-on-none” – meaning it’s asynchronous, and there is no moderator present. A lot has been written about the effects of group-think in focus groups, where an ‘alpha’ respondent will influence others. Similarly, the confessional style of the Fabric methodology enables what researchers have called the “online disinhibition effect” where respondents are more open to express themselves because there is no fear of disagreement or conflict with a moderator. For consumers, communicating asynchronously via video is a very comfortable and common medium

Fabric POV

Our short answer when clients ask us about the right sample size is to go with 30-50 respondents, but the context as outlined in the “Fabric POV” section above guides us beyond.

Follow up questions are welcome: please email tom@fabric.is.

Back to all pro tips
Posted on

pro-tips-fabric-video

4 Tips for Getting the Most out of Your Fabric Video

What’s the best way to go through video responses and glean insights? Tom Bassett, founder of Fabric & mindswarms, discusses best practices for extracting relevant data and weaving powerful stories to share within your organization.

Mobile video surveys provide powerful first-person accounts that speak customer truths. But how do you get the most from your data? Using his more than 20 years of experience working with companies like Microsoft, Virgin America, and Nike, Tom Bassett shares 4 tips to help you effectively analyze your data and draw conclusions.

Reviewing Results

When analyzing your mobile video surveys, there are a couple of things to look for. First, you’ll want to identify patterns and themes. What are people identifying with? What are the recurring problems or issues with the product, service, or experience? Come up with a list, aiming for no more than 5-10 patterns. Once you’ve identified these, it’s important to run them through your brand filter. Ask yourself: How are the insights related to my brand? Which are the most relevant for you (versus insights that are not related to my brand)?

During this stage, it’s also important to look for original insights – outliers. For example, what’s something you didn’t expect the respondents to say? What’s something unique you noticed, when seen through the eyes of the consumer? Since it’s very hard for brands to differentiate these days, the off-speed pitch is often where you find the interesting angles.

Organizing Results

Next, you’ll want to bracket your insights into two main buckets: problems and opportunities. This is a relevant way to share insights back internally at your organization. Senior leadership teams often want to help solve problems – but they also want to understand where the potential zones of opportunity are to help grow. Are the respondents articulating a problem or need that hasn’t been met? Have they mentioned a totally new idea? An expansion of idea?

Identifying Story

This is an important step because although it’s great to have lots of insight, it’s essential to find a focal point. Ask yourself: What is the overarching story? How do you articulate this? Try to hone in and articulate one story. Philosophically, we like to see things through the eyes of consumers. For example, for a Yahoo Personals project: “Women don’t want to let go of the idea that Fate played a role in finding the partner of their dreams.” Once you determine THE story, you’ll want to come up with chapters, or building blocks that help build that story. Ask yourself: What are we trying to teach or tell people? What’s the big reveal? In simple terms, there is a beginning, middle and end. The middle is usually the reveal (the point of tension, the climax) while the beginning introduces it, and the end wraps it up.

Sharing Results

When you give your presentation to colleagues, use the videos to really engage them! Videos lift heads, because there is sight and sound and motion. So, make sure to show video clips of respondents in order to get their attention. It’s worth it to put together a 2-3 minute series of clips in order to share the data in the most powerful way possible. But don’t use video to be a surrogate for PowerPoint; use video to tell a compelling story. Otherwise, if video just lists points as opposed to adding up to something singular, viewers will be confused. (One additional tidbit: we also like to sprinkle in single Fabric clips throughout a debrief deck, to help keep things lively. These clips can be links to the study matrix, or videos can be downloaded and placed into the presentation).

Back to all pro tips
Posted on

pro-tips-fabric-traditional-research

4 ways Fabric works with traditional qualitative research

Recruiting for ethnography.

Rather than show up at a respondent’s home with an Excel spreadsheet and fingers crossed the participant is good, Fabric has been used to recruit in advance for ethnographies. Not only does it help identify cream-of-the-crop candidates, but it also informs the research process early on, and can be used to sharpen discussion guides.

Pre-in field.

What we have often done is a wave of Fabric research in advance of doing any kind of field work (focus groups, IDIs, in-homes). A lot of clients are anxious to get projects moving quickly, and instead of waiting two weeks to begin traditional qual research, Fabric can gather insight within days from a very broad geography to begin to inform the project.

Post in-field.

Fabric has been used as a way to bounce ideas off consumers after the traditional qualitative field work has been completed. So, for example, on a project with product designers, sketches of ideas inspired on the road were bounced off Fabric respondents after the initial wave of research had been completed to help validate a direction.

Extending reach of focus groups.

In most countries, the tendency is to conduct traditional qualitative research in major markets because that’s where the lion’s share of volume comes from. But Fabric has been used in a remarkably quick way to access consumers in every level of markets to help balance out a more urban skew to the recruitment. So, for example, in the US, rather than hit the usual NY/LA/Chicago markets, sprinkling in a nationally representative wave of Fabric recruits helps provide a more representative picture of consumers nationally.

Back to all pro tips
Posted on

pro-tips-survey-questions

How to Write Effective Mobile Video Survey Questions

The art of asking questions

In any and every study, elements of great research, creativity and innovation revolve around asking the right questions. Since the Fabric methodology is unique, below is a set of best practices for asking questions on the platform in order to reach the “unlock.”

Start broad, then get specific

We are big proponents of trying to understand consumers in the broadest possible context, including culturally. So beginning studies with a sense for how they think the culture is changing is a great way to anchor their later insights within a cultural context; so much consumer behavior is shaped by much greater forces than product features.

Ask specifically vague questions

It’s tempting to go straight at an issue or a problem. But often times, it’s best to understand where THEY will take the story. So, for example, if you ask whether they like cars – or not – you will understand whether they like cars. But if you ask them to explain their relationship with cars, you’re never sure where they will take that story. And understanding the broader context of their automotive relationship may be much more insightful than a specific like/dislike question.

Ask very pointed questions

Seemingly contrary to asking specifically vague questions is to flip the script and be VERY direct and pointed. Sometimes even revealing the real question at hand can help consumers provide highly pointed responses.

Be provocative

Video is the most emotional medium, so embracing what it captures best by provoking (without, of course, insulting) can be an effective tactic. Prompting them with statements, or published articles will allow them to react. For example, linking Millennials to an article about their perceived attitudes and behaviors and asking respondents to weigh in on whether they agree or disagree can provide deeper levels of understanding.

Use polarizing questions

Respondents are opinionated. Take advantage of their strong opinions by asking them what they love or what they hate, especially if an emotional response is what you’re looking for. For those who are less opinionated, forcing them to choose left/right or high/low – basically making them choose – helps clarify which side of the divide they are on.

Ask why

The thought processes behind the decisions that people make are perhaps even more important than the decision themselves. Have the respondents explain their perspective and why they do things. Asking “why” seems perhaps too elemental sometimes, but by asking the obvious why question, it can help unearth new ways of understanding consumers (e.g. why do you run? Why do you shop?).

Get respondents into relevant space

The big advantage of using mobile video (and webcams) is that you can be in the respondent’s space. Have them bring you to the environment that makes the most sense for your mobile video survey. We have had respondents record from their kitchen, bathroom (for a shaving study), garage, pantry, bedroom (for a closet dive), den (for home entertainment studies), and more.

Don’t cram 5 questions into 1

Imagine we toss you a tennis ball. Easy enough to catch, right? But what if we toss you three? Not so easy. Stick to one question (or tennis ball). At Fabric, respondents have one minute to respond, and you have 200 characters to write your question. You don’t want respondents to spend the entire minute just listing off things or juggling their focus. A good stress test is to make sure you are only using one question mark – at most, two.

Tug at respondents’ emotions

The best insight comes when people talk about things that they really care about, whether it is something that they love or a secret pet peeve of theirs. Ask questions that aim at eliciting these emotions. To that end, for a pet food study, we had consumers introduce us to their pet in the first video; people projected voices, personalities and deep emotion in the first response. Similarly, for a study about Millennial women and cleaning, we had them hold up a photo of their mothers in the first reply, and it choked some of them up!

Use their language, not yours

Use language that the respondents are comfortable with, and would use if they were talking to a friend. For instance, a respondent might not know what an “asset” is.

Allow for open-ended questions

We’ve found that respondents usually have some additional thoughts at the end of the survey that haven’t been addressed by any questions. Giving them the freedom to share these thoughts with you can lead to even more novel insights.

Refine on the fly

Time permitting, what we like to do is have the first few participants respond to see how they interpret the questions. Questions can be amended as the study progresses, so sometimes even slight alterations to questions can be help drill down to a deeper level.

Be creative

Put your respondents in hypothetical situations, use similes and metaphors, or ask a question that is completely “out there.” The more creative your question is, the more creative (and interesting) your responses will be.

Back to all pro tips
Posted on

pro-tips-methodologies

4 Methodologies to use with Fabric Survey Videos

Perceptions & Attitudes

Framed as a video selfie talking head, Perceptions & Attitudes captures people’s facial expressions with a proximity closer than any other methodology can provide (literally, and arm’s length away).

The beauty of Perceptions & Attitudes is there is no moderator bias, and no possibility for group-think that can be common problem in focus groups.

Micro facial expressions, including eyebrow raises, mouth movements, involuntary twitches, voice intonation and more help establish their response authenticity and candor.

Given their comfort level with speaking into their own mobile devices or webcams, the following is a list of strategies for getting the best out of Perceptions & Attitudes:

  • Get personal: they are in their comfort zone, usually alone, and amazingly open to questions that probe deeply into their personal experience and/or feelings. We have asked people about chronic illness (delicately, of course), intimate wear (in relation to their body image), impotence (you’d be surprised how comfortable men are on the topic) and more; and are constantly inspired by how open people are via mobile video
  • Be provocative: ask them about their deepest fears, their greatest achievements, their trigger points for anger, injustice…even politics! Provoking – without insulting them, of course – helps elicit more emotionally-charged replies
  • Benchmark: ask them to compare and contrast things like brand associations, competing product perceptions, standards of customer service against the industry norms; or even the ideal
  • Be emotional: while some respondents are better than others at answering questions about emotion, don’t shy away from literally asking them about emotions surrounding a topic. We’ve even had brand loyalists write a love letter to a brand and read the letters on camera
  • Allow them to dream: one of our favorite questions is to have them suspend disbelief, and describe the ideal . By understanding the dream, you will be better able to gauge if/how you or your brand/product/service lives up to their ideal

Show & Tell

Show + Tell is a way to see into people home environments: pantry, closet, den, fridge, entertainment system, home office….you name it.

In our experience, having people show objects – or processes – and talk about them tends to not only authenticate them as users, but also makes them much more animated and articulate because they are touching objects, or pointing out parts of a journey.

We define Show + Tell as something that happens within or around the home (versus Retail + Events which can take place outside the home).

Show + Tell has been used for a wide range of projects including clothing (show us your favorite sports bras, and explain how brands and product features vary), cars (walk around your car and tell us about brand/features/ design), food (fridge dive to understand brand assortment; prepare meals to understand process), entertainment (TV and audio ecosystem), furniture (favorite room, biggest villain in the house), laundry journey, shaving experience, and kitchen (storage container lids).

To get the best out of Show + Tell:

  • Be specific: ask them to show you their favorite X, or top 5 articles of Y (e.g. show us your 5 favorite handbags)
  • Walk-throughs: ask them to walk you through a process – laundry, for example, to understand the Before, During and After insights (sometimes breaking it out into multiple questions)
  • Pan-arounds: or closet dives, or home entertainment ecosystems, have them pan over a variety of objects and explain what they are, why they have them and note brands (e.g. we had a big tech co whose designer wasn’t born in the US and wanted to understand what a college dorm room looked like)
  • Expect variable video: if you request consumers to show their laundry journey, for example, the video itself might be a little shaky as they pan around and explain things, or walk from room to room. We can mitigate AV issues with instructions, but lighting, sound and video quality may vary if they are asked to move from room to room while filming, from light to dark, or spinning the smartphone around to show something
  • Involve a partner: for a study for baby wash basins, respondents had their partner hold their phone and record moments where both of their hands were occupied

Prompt & React

Getting consumers to respond to prompts can be accomplished on mindswarms digitally, or physically. Digitally, by attaching a link to a study, respondents can view PDFs, images, web sites, video, UX, UI…basically anything you want them to provide unvarnished reactions to. To date, we have prompted consumers with links to early stage concepts (designer sketches, advertising territories, brand positioning), work-in-progress (ads in development, taglines, potential product names) and finished assets (existing TV commercials, websites, digital products, print ads).

  • Similarly, we have sent product to people’s homes for a number of objectives:
  • Understand the out of box experience
  • Gather feedback on packaging
  • Get reactions to messaging and positioning
  • Conduct a home use test (prepare food; try out makeup, try on new prototype shoes) In terms of best practices for working with Prompt & React, the following is a loose set of guidelines:

In terms of best practices for working with Prompt & React, the following is a loose set of guidelines:

    • Keep it short: if a PDF that is attached contains too many pages, images or words, people will tune out
    • Think about it as stimulus: sometimes, their literal reactions to the artifact aren’t as important as the reactions they elicit – what you’re really looking for is how the stimulus prompted them to reveal something interesting or new…so it doesn’t always have to be stimulus you are literally testing that elicits a great insight (for example, get them to respond to an article about their generation and explain how they feel)
    • Plan for shipping: if it’s an actual product you want sent to consumers, shipping will be your responsibility (not mindswarms’)
    • Need 1 or two more

Missions & Events

A lot of retail experiences require no assistance on the shop floor (e.g. headphones, QSR). So hearing from consumers in a retail environment as they first enter the store, look for sections, search for products, compare competing options, evaluate packaging, look at signage, navigate the aisles, and ultimately choose a specific brand or product can be very insightful. Especially as more and more shopping and buying shifts online.

Retail & Events has been used to have consumers shop for e-readers at Big Box consumer electronics stores, do a walk-through of a car buying journey, sample food at QSR, do competitor store checks, shop a new category in a store, evaluate drive-thru menu boards, provide feedback on the inclusiveness of in-store marketing messaging and more

In terms of pro tips for Retail & Events:

  • It’s its own study: while we can – and do – include Perceptions + Attitudes, Show + Tell, and Prompt + React in one study, getting folks to go to retail requires a dedicated study devoted to the retail visit
  • Incentives need to be higher: we typically pay $50 for consumers to answer 10 questions, but if they need to travel to a retail store and/or buy something, we often sweeten the kitty to compensate them for their time, effort and travel
  • Studies can take longer than normal:
    we promise <7 days for US studies and <14 days for international, but if the study requires people to do a retail visit, we like to give them a little more time to complete, especially if weekends are the only time they might be able to fit in a destination trip to – say – a mall
Back to all pro tips