pro-tips-qualitative-sample-sizes

POV: Qualitative Sample Sizes

Purpose

The purpose of this content is to provide a point of view on the ideal sample size for a qualitative market research study, because that’s one of the most common questions even the most seasoned researcher asks us.

Industry norms

Adequate sample sizes have been debated extensively in the market research industry for many years, in part because quantitative sample sizes are statistically easier to measure. Often, quantitative research frameworks get applied to qualitative research because quant norms are concrete (e.g +/- variance), whereas qualitative sample sizes are difficult to prove, mathematically.

A number of studies have been published over the years on the topic of adequate sample sizes for qualitative research. Here are a few:

    • Creswell, Glaser, Morse – all pretty much agree that sample sizes of 30-50 respondents reaches a point of saturation, where adding more respondents does not significantly alter the findings
    • Springer – Springer puts forth the argument that anywhere from 5-50 is adequate, but that 25-30 is considered to be the right number
    • InterQ – recommends 20-30, or even as few as 10

Fabric POV

Qualitative research is more art than science. Bill Bowerman (Nike co-founder) tested his shoe designs on a relatively small sample size, yet was able to scale the innovation to millions of people, in large part because it was the right sample of runners.

Typically, clients turn to qualitative methods to understand meaning – the How and the Why. 

So while we believe there are minimum sample sizes that should be employed in qualitative research, there are some critical variables in the equation we consider:

    • where you are in the process; in our opinion, earlier stage projects (e.g. exploratory product innovation) can employ smaller sample sizes, because more iteration and development will be conducted as the project moves along
    • the business impact: we feel that the higher the business impact of the research, the greater the sampling (qualitative and quantitative combined) should be
    • geography: it can be as easy as domestic and international, but one critical question we consider is how broad the target audience is, geographically
    • research design: how the study is designed can have a huge impact on results (e.g. which questions to ask in what order)
    • research vendor: when you buy a qualitative research resource, you are buying a filter (like a water or air filter). One important variable is how equipped that resource is, down to the people, to field the study and interpret the results
    • quality of recruit: back to the Bowerman example, it’s very important that the people in the study be the right people; beyond technically whether they qualify, we ask ourselves whether the people we interview “feel” like the right consumer. And we will over-index the input of those whom we deem more influential (for example, a starter on an AAU basketball team in Brooklyn may be more insightful than a practice player on a D3 team in the suburbs)
    • analysis: who is interpreting the data is highly important. 
    • methodology: there are a number of qualitative techniques to choose from, including IDIs (individual interviews), focus groups, in-homes, friendship pairs, small group interviews, intercepts, observational research, ethnographies, and digital (online, mobile). In Fabric’s case, the methodology is unique in that it’s effectively “1-on-none” – meaning it’s asynchronous, and there is no moderator present. A lot has been written about the effects of group-think in focus groups, where an ‘alpha’ respondent will influence others. Similarly, the confessional style of the Fabric methodology enables what researchers have called the “online disinhibition effect” where respondents are more open to express themselves because there is no fear of disagreement or conflict with a moderator. For consumers, communicating asynchronously via video is a very comfortable and common medium

Fabric POV

Our short answer when clients ask us about the right sample size is to go with 30-50 respondents, but the context as outlined in the “Fabric POV” section above guides us beyond.

Follow up questions are welcome: please email tom@fabric.is.

pro-tips-creative-cases-mindswarms

Creative and inspiring use cases of Fabric

The following is a short list of ways in which we and/or our clients have employed the platform creatively to extract new and unusual insights.

Baby wash basin

For a leading brand of baby strollers that was looking into extending their product footprint into the baby washbasin space, we had parents of kids <1 years old wash their babies on camera (their privates were covered with a washcloth) and talk about the experience. In this instance, the partner held the phone to capture the experience. There is literally no other way to do this: imagine sending a videographer and interviewer into their home and asking them to wash their baby while you film it? Right; we don’t think so either.

Prepare and eat food

For one of the country’s leading packaged food innovation companies, we shipped a potential new product to consumers’ homes, then had them record their out of box experience, the preparation, and finally tasting of the product. Interestingly, while their initial impressions bordered on repulsion (“it looks like rabbit poo!”), the taste of the product was incredibly well received, providing the client with insight on how to tweak the product extrusion.

Write a love letter to the brand and read it aloud

For one of the world’s biggest furniture brands and retailers, we had loyalists write a love letter to the brand, and read it aloud on camera. It was for an exercise to get at what the core of the brand stood for, so having consumers – in their own words – craft and read a love letter on camera, it helped surface some of the deeper emotional connections they had with the brand.

Go to a drive thru

For a major QSR brand that was looking at re-designing its drive-thru menu board, we had consumers of the brand report on the before, during and after of the drive-thru experience at their stores. Recording video from inside their cars in the moment, the client was able to get a first-hand look at how and where the messaging was connecting and missing.

Shop for spray paint

For a global ad agency pitching a new potential client, we had consumes shop for the client’s product at a chain of major big box stores. Because like many categories, the product and landscape was crowded and competitive, so having consumers report on their thoughts on the buying process as well as packaging and displays helped drive a deeper understanding of how hard it would be for the brand to stand out.

Show us your favorite sports bras

For a project to help re-launch and re-position a line of sports bras for a leading brand of sports footwear and apparel, we had women show us their collection of sports bras and talk about them. Any time people hold something and talk about it, it not only helps validate them as the right consumers, but it also makes them more animated and articulate in their responses.

Step outside of a party

For a project with truth (anti smoking), we had teens step outside from a party they were attending and record their responses to key questions that would arise for them in a party situation. Although the lighting quality was variable (a number of the responses were quite dark), the insights were powerful because they were so close to the moment (specifically a situation loaded with peer pressure).

SONOS diary

To understand how consumers’ relationship with music changed in the home using SONOS, we shipped speakers to people all over the world. And had them record their thoughts over a period of one month.

Laundry journey

To understand potential product extension zones of opportunity for a major European appliance brand, we had consumers deconstruct their before, during and after laundry journeys.

Political advertising

Early on in the Trump campaign when the first TV commercial launched, we were curious to understand if/how the messaging was resonating and connecting. While we are politically neutral, we predicted early on that he had tapped into a strong emotional undercurrent in the country, and our founder was quoted as such in Mother Jones magazine.

Introduce us to your cat

For a pet food study for one of the country’s biggest and best known brands, we had respondents introduce us to their cats in the first response. The technique helped not only validate that owners did indeed have cats, it made respondents more emotionally engaged, and helped add color/texture to their insights.

Show us the biggest villain in your home

Working with one of the world’s largest furniture brands and retailer, we had consumers show us what they deemed to be “the biggest villain in the home” with the goal being to understand how that connected to the American Dream.

Home use test

We have shipped food products, SONOS speakers, shoe prototypes and make-up to people’s homes and had them record everything from their “out of box” experience to preparation/set up, use and even removal of make-up.

pro-tips-fabric-video

4 Tips for Getting the Most out of Your Fabric Video

What’s the best way to go through video responses and glean insights? Tom Bassett, founder of Fabric & mindswarms, discusses best practices for extracting relevant data and weaving powerful stories to share within your organization.

Mobile video surveys provide powerful first-person accounts that speak customer truths. But how do you get the most from your data? Using his more than 20 years of experience working with companies like Microsoft, Virgin America, and Nike, Tom Bassett shares 4 tips to help you effectively analyze your data and draw conclusions.

Reviewing Results

When analyzing your mobile video surveys, there are a couple of things to look for. First, you’ll want to identify patterns and themes. What are people identifying with? What are the recurring problems or issues with the product, service, or experience? Come up with a list, aiming for no more than 5-10 patterns. Once you’ve identified these, it’s important to run them through your brand filter. Ask yourself: How are the insights related to my brand? Which are the most relevant for you (versus insights that are not related to my brand)?

During this stage, it’s also important to look for original insights – outliers. For example, what’s something you didn’t expect the respondents to say? What’s something unique you noticed, when seen through the eyes of the consumer? Since it’s very hard for brands to differentiate these days, the off-speed pitch is often where you find the interesting angles.

Organizing Results

Next, you’ll want to bracket your insights into two main buckets: problems and opportunities. This is a relevant way to share insights back internally at your organization. Senior leadership teams often want to help solve problems – but they also want to understand where the potential zones of opportunity are to help grow. Are the respondents articulating a problem or need that hasn’t been met? Have they mentioned a totally new idea? An expansion of idea?

Identifying Story

This is an important step because although it’s great to have lots of insight, it’s essential to find a focal point. Ask yourself: What is the overarching story? How do you articulate this? Try to hone in and articulate one story. Philosophically, we like to see things through the eyes of consumers. For example, for a Yahoo Personals project: “Women don’t want to let go of the idea that Fate played a role in finding the partner of their dreams.” Once you determine THE story, you’ll want to come up with chapters, or building blocks that help build that story. Ask yourself: What are we trying to teach or tell people? What’s the big reveal? In simple terms, there is a beginning, middle and end. The middle is usually the reveal (the point of tension, the climax) while the beginning introduces it, and the end wraps it up.

Sharing Results

When you give your presentation to colleagues, use the videos to really engage them! Videos lift heads, because there is sight and sound and motion. So, make sure to show video clips of respondents in order to get their attention. It’s worth it to put together a 2-3 minute series of clips in order to share the data in the most powerful way possible. But don’t use video to be a surrogate for PowerPoint; use video to tell a compelling story. Otherwise, if video just lists points as opposed to adding up to something singular, viewers will be confused. (One additional tidbit: we also like to sprinkle in single Fabric clips throughout a debrief deck, to help keep things lively. These clips can be links to the study matrix, or videos can be downloaded and placed into the presentation).

pro-tips-fabric-traditional-research

4 ways Fabric works with traditional qualitative research

Recruiting for ethnography.

Rather than show up at a respondent’s home with an Excel spreadsheet and fingers crossed the participant is good, Fabric has been used to recruit in advance for ethnographies. Not only does it help identify cream-of-the-crop candidates, but it also informs the research process early on, and can be used to sharpen discussion guides.

Pre-in field.

What we have often done is a wave of Fabric research in advance of doing any kind of field work (focus groups, IDIs, in-homes). A lot of clients are anxious to get projects moving quickly, and instead of waiting two weeks to begin traditional qual research, Fabric can gather insight within days from a very broad geography to begin to inform the project.

Post in-field.

Fabric has been used as a way to bounce ideas off consumers after the traditional qualitative field work has been completed. So, for example, on a project with product designers, sketches of ideas inspired on the road were bounced off Fabric respondents after the initial wave of research had been completed to help validate a direction.

Extending reach of focus groups.

In most countries, the tendency is to conduct traditional qualitative research in major markets because that’s where the lion’s share of volume comes from. But Fabric has been used in a remarkably quick way to access consumers in every level of markets to help balance out a more urban skew to the recruitment. So, for example, in the US, rather than hit the usual NY/LA/Chicago markets, sprinkling in a nationally representative wave of Fabric recruits helps provide a more representative picture of consumers nationally.

pro-tips-follow-ups

pro-tips-survey-questions

How to Write Effective Mobile Video Survey Questions

The art of asking questions

In any and every study, elements of great research, creativity and innovation revolve around asking the right questions. Since the Fabric methodology is unique, below is a set of best practices for asking questions on the platform in order to reach the “unlock.”

Start broad, then get specific

We are big proponents of trying to understand consumers in the broadest possible context, including culturally. So beginning studies with a sense for how they think the culture is changing is a great way to anchor their later insights within a cultural context; so much consumer behavior is shaped by much greater forces than product features.

Ask specifically vague questions

It’s tempting to go straight at an issue or a problem. But often times, it’s best to understand where THEY will take the story. So, for example, if you ask whether they like cars – or not – you will understand whether they like cars. But if you ask them to explain their relationship with cars, you’re never sure where they will take that story. And understanding the broader context of their automotive relationship may be much more insightful than a specific like/dislike question.

Ask very pointed questions

Seemingly contrary to asking specifically vague questions is to flip the script and be VERY direct and pointed. Sometimes even revealing the real question at hand can help consumers provide highly pointed responses.

Be provocative

Video is the most emotional medium, so embracing what it captures best by provoking (without, of course, insulting) can be an effective tactic. Prompting them with statements, or published articles will allow them to react. For example, linking Millennials to an article about their perceived attitudes and behaviors and asking respondents to weigh in on whether they agree or disagree can provide deeper levels of understanding.

Use polarizing questions

Respondents are opinionated. Take advantage of their strong opinions by asking them what they love or what they hate, especially if an emotional response is what you’re looking for. For those who are less opinionated, forcing them to choose left/right or high/low – basically making them choose – helps clarify which side of the divide they are on.

Ask why

The thought processes behind the decisions that people make are perhaps even more important than the decision themselves. Have the respondents explain their perspective and why they do things. Asking “why” seems perhaps too elemental sometimes, but by asking the obvious why question, it can help unearth new ways of understanding consumers (e.g. why do you run? Why do you shop?).

Get respondents into relevant space

The big advantage of using mobile video (and webcams) is that you can be in the respondent’s space. Have them bring you to the environment that makes the most sense for your mobile video survey. We have had respondents record from their kitchen, bathroom (for a shaving study), garage, pantry, bedroom (for a closet dive), den (for home entertainment studies), and more.

Don’t cram 5 questions into 1

Imagine we toss you a tennis ball. Easy enough to catch, right? But what if we toss you three? Not so easy. Stick to one question (or tennis ball). At Fabric, respondents have one minute to respond, and you have 200 characters to write your question. You don’t want respondents to spend the entire minute just listing off things or juggling their focus. A good stress test is to make sure you are only using one question mark – at most, two.

Tug at respondents’ emotions

The best insight comes when people talk about things that they really care about, whether it is something that they love or a secret pet peeve of theirs. Ask questions that aim at eliciting these emotions. To that end, for a pet food study, we had consumers introduce us to their pet in the first video; people projected voices, personalities and deep emotion in the first response. Similarly, for a study about Millennial women and cleaning, we had them hold up a photo of their mothers in the first reply, and it choked some of them up!

Use their language, not yours

Use language that the respondents are comfortable with, and would use if they were talking to a friend. For instance, a respondent might not know what an “asset” is.

Allow for open-ended questions

We’ve found that respondents usually have some additional thoughts at the end of the survey that haven’t been addressed by any questions. Giving them the freedom to share these thoughts with you can lead to even more novel insights.

Refine on the fly

Time permitting, what we like to do is have the first few participants respond to see how they interpret the questions. Questions can be amended as the study progresses, so sometimes even slight alterations to questions can be help drill down to a deeper level.

Be creative

Put your respondents in hypothetical situations, use similes and metaphors, or ask a question that is completely “out there.” The more creative your question is, the more creative (and interesting) your responses will be.

pro-tips-methodologies

4 Methodologies to use with Fabric Survey Videos

Perceptions & Attitudes

Framed as a video selfie talking head, Perceptions & Attitudes captures people’s facial expressions with a proximity closer than any other methodology can provide (literally, and arm’s length away).

The beauty of Perceptions & Attitudes is there is no moderator bias, and no possibility for group-think that can be common problem in focus groups.

Micro facial expressions, including eyebrow raises, mouth movements, involuntary twitches, voice intonation and more help establish their response authenticity and candor.

Given their comfort level with speaking into their own mobile devices or webcams, the following is a list of strategies for getting the best out of Perceptions & Attitudes:

  • Get personal: they are in their comfort zone, usually alone, and amazingly open to questions that probe deeply into their personal experience and/or feelings. We have asked people about chronic illness (delicately, of course), intimate wear (in relation to their body image), impotence (you’d be surprised how comfortable men are on the topic) and more; and are constantly inspired by how open people are via mobile video
  • Be provocative: ask them about their deepest fears, their greatest achievements, their trigger points for anger, injustice…even politics! Provoking – without insulting them, of course – helps elicit more emotionally-charged replies
  • Benchmark: ask them to compare and contrast things like brand associations, competing product perceptions, standards of customer service against the industry norms; or even the ideal
  • Be emotional: while some respondents are better than others at answering questions about emotion, don’t shy away from literally asking them about emotions surrounding a topic. We’ve even had brand loyalists write a love letter to a brand and read the letters on camera
  • Allow them to dream: one of our favorite questions is to have them suspend disbelief, and describe the ideal . By understanding the dream, you will be better able to gauge if/how you or your brand/product/service lives up to their ideal

Show & Tell

Show + Tell is a way to see into people home environments: pantry, closet, den, fridge, entertainment system, home office….you name it.

In our experience, having people show objects – or processes – and talk about them tends to not only authenticate them as users, but also makes them much more animated and articulate because they are touching objects, or pointing out parts of a journey.

We define Show + Tell as something that happens within or around the home (versus Retail + Events which can take place outside the home).

Show + Tell has been used for a wide range of projects including clothing (show us your favorite sports bras, and explain how brands and product features vary), cars (walk around your car and tell us about brand/features/ design), food (fridge dive to understand brand assortment; prepare meals to understand process), entertainment (TV and audio ecosystem), furniture (favorite room, biggest villain in the house), laundry journey, shaving experience, and kitchen (storage container lids).

To get the best out of Show + Tell:

  • Be specific: ask them to show you their favorite X, or top 5 articles of Y (e.g. show us your 5 favorite handbags)
  • Walk-throughs: ask them to walk you through a process – laundry, for example, to understand the Before, During and After insights (sometimes breaking it out into multiple questions)
  • Pan-arounds: or closet dives, or home entertainment ecosystems, have them pan over a variety of objects and explain what they are, why they have them and note brands (e.g. we had a big tech co whose designer wasn’t born in the US and wanted to understand what a college dorm room looked like)
  • Expect variable video: if you request consumers to show their laundry journey, for example, the video itself might be a little shaky as they pan around and explain things, or walk from room to room. We can mitigate AV issues with instructions, but lighting, sound and video quality may vary if they are asked to move from room to room while filming, from light to dark, or spinning the smartphone around to show something
  • Involve a partner: for a study for baby wash basins, respondents had their partner hold their phone and record moments where both of their hands were occupied

Prompt & React

Getting consumers to respond to prompts can be accomplished on mindswarms digitally, or physically. Digitally, by attaching a link to a study, respondents can view PDFs, images, web sites, video, UX, UI…basically anything you want them to provide unvarnished reactions to. To date, we have prompted consumers with links to early stage concepts (designer sketches, advertising territories, brand positioning), work-in-progress (ads in development, taglines, potential product names) and finished assets (existing TV commercials, websites, digital products, print ads).

  • Similarly, we have sent product to people’s homes for a number of objectives:
  • Understand the out of box experience
  • Gather feedback on packaging
  • Get reactions to messaging and positioning
  • Conduct a home use test (prepare food; try out makeup, try on new prototype shoes) In terms of best practices for working with Prompt & React, the following is a loose set of guidelines:

In terms of best practices for working with Prompt & React, the following is a loose set of guidelines:

    • Keep it short: if a PDF that is attached contains too many pages, images or words, people will tune out
    • Think about it as stimulus: sometimes, their literal reactions to the artifact aren’t as important as the reactions they elicit – what you’re really looking for is how the stimulus prompted them to reveal something interesting or new…so it doesn’t always have to be stimulus you are literally testing that elicits a great insight (for example, get them to respond to an article about their generation and explain how they feel)
    • Plan for shipping: if it’s an actual product you want sent to consumers, shipping will be your responsibility (not mindswarms’)
    • Need 1 or two more

Missions & Events

A lot of retail experiences require no assistance on the shop floor (e.g. headphones, QSR). So hearing from consumers in a retail environment as they first enter the store, look for sections, search for products, compare competing options, evaluate packaging, look at signage, navigate the aisles, and ultimately choose a specific brand or product can be very insightful. Especially as more and more shopping and buying shifts online.

Retail & Events has been used to have consumers shop for e-readers at Big Box consumer electronics stores, do a walk-through of a car buying journey, sample food at QSR, do competitor store checks, shop a new category in a store, evaluate drive-thru menu boards, provide feedback on the inclusiveness of in-store marketing messaging and more

In terms of pro tips for Retail & Events:

  • It’s its own study: while we can – and do – include Perceptions + Attitudes, Show + Tell, and Prompt + React in one study, getting folks to go to retail requires a dedicated study devoted to the retail visit
  • Incentives need to be higher: we typically pay $50 for consumers to answer 10 questions, but if they need to travel to a retail store and/or buy something, we often sweeten the kitty to compensate them for their time, effort and travel
  • Studies can take longer than normal:

we promise <7 days for US studies and <14 days for international, but if the study requires people to do a retail visit, we like to give them a little more time to complete, especially if weekends are the only time they might be able to fit in a destination trip to – say – a mall

case-studies-launch-new-product

How Mobile Video Surveys Helped DINE Launch a New Product

When food innovation company DINE needed to make a snap decision about whether to introduce a new product to consumers, they turned to mindswarms. The feedback they got was invaluable, and quick. (The consumers’ facial expressions alone speak volumes.)

“Basically they would think that maybe I was racist because I’m speaking out about it [police brutality]. It’s scary that there’s a lot of people out there that can’t even go to the store at night without worrying about dying, and there’s people who can get pulled over, and pull a gun out, and maybe because of their skin color or their background they won’t even die…”

Much has been written about the benefits of using mobile video surveys to glean consumer insights. This method empowers respondents to speak freely, and captures human emotions through first-person accounts in an environment that is familiar to them. Also, it’s quicker than field research.

That’s why when DINE needed to make a fast decision about whether to launch a new product, they chose to use mobile video surveys. The process was so efficient. They sent packages of the new product to 30 households around the U.S., and consumers recorded their reactions using their web cams as they opened the package, cooked the food and tasted it for the first time.

DINE was struck by how much insightful information they got, not only from the respondents’ words, but by seeing their facial expressions and physical environment. The qualitative data was rich and shaped their next steps: Despite serious initial apprehension about how the product looked, the consumers all agreed that the product tasted great. DINE knew exactly what they need to do to overcome this and make the launch successful. The product was green-lit.

How Mobile Video Surveys Helped DINE Launch a New Product - Fabric
When food innovation company DINE needed to make a snap decision about whether to introduce a new product to consumers, they turned to mindswarms. The feedback they...
Millennials & Home Cleaning - Fabric
In a recent study, mindswarms set out to understand unique generational considerations in how Millennial women relate to home cleaning, home cleaning brands, and home cleaning product...
Amplify Your Pitch with Mobile Video Surveys - Fabric
4 days away from a major pitch, BBDO knew that having authentic & timely consumer video insights would amplify their pitch. The result: they won.
How to Seal the Deal with Mobile Video Surveys - Fabric
What happens when you need consumer insights within a matter of days? Check out how Fabric helped a leading agency craft video surveys to win new business.
mindswarms Partners with Carnegie Mellon HCI Masters Class - Fabric
In a pilot study at Carnegie Mellon University, students enrolled in Human Computing Interaction Masters program used mindswarms video mobile surveys to get feedback on their idea...
Mobile Video Ethnography Dives Into the Teenage Mind - Fabric
What shapes the identities of teens today? California ad agency BSSP wanted to reach beyond data-driven assumptions to find out, going deep into the minds of Generation...
How micro moments are reshaping the travel customer journey - Fabric
There’s a lot of planning that goes into traveling. And people increasingly turn to devices for help. Here, Think with Google (powered by mindswarms) reveals the consumer...
Mobile Video Surveys Reframe Global Qual - Fabric
Rob Klingensmith, Chief Strategic Officer of Marcel Worldwide, worked with mindswarms to collect consumer insights in 5 different countries for Luxottica, makers of the world’s iconic Ray-Ban...
How mobile has redefined the consumer decision journey - Fabric
The consumer journey, from I-need-some-ideas through I-want-to-buy-it, is increasingly guided by the smartphone.This means retailers need to show up in a useful way online in order to...
Dr. Lautman - Professor at Wharton’s MBA program - on Mobile Video Surveys - Fabric
Wharton MBA adjunct professor Dr. Martin Lautman appreciates the benefits of big data. But when it comes to diving deeper in order to understand consumer emotion and...

case-studies-millenials

Millennials & Home Cleaning

In a recent study, mindswarms set out to understand unique generational considerations in how Millennial women relate to home cleaning, home cleaning brands, and home cleaning product purchasing. What we didn’t anticipate? That we’d open up a Pandora’s Box of emotionally deep insights. You can watch a video clip here.

“Basically they would think that maybe I was racist because I’m speaking out about it [police brutality]. It’s scary that there’s a lot of people out there that can’t even go to the store at night without worrying about dying, and there’s people who can get pulled over, and pull a gun out, and maybe because of their skin color or their background they won’t even die…”

Research Objective

To understand the emotions, pressures, and motivations related to household cleaning. By gaining a deeper understanding of the the modern young woman’s relationship to cleaning, we can better gauge what types of products and concepts would appeal most. One interesting technique we employed was to have women show us a photo of their Mom in the first response, to dial up the emotional intensity of their responses.

Target Audience

  • National US sample
  • 12 states
  • 14 cities
  • Ages 25 – 40
  • Female
  • Mix of ethnicities and socio-economic backgrounds

We Learned:

While we anticipated interesting results given the nature of the study and the research design, what we didn’t see coming was that the study became a form of mother/daughter therapy as told through the lens of home cleaning. Millennial women’s cleaning rituals, habits and schedules opened up a Pandora’s box of deep memories, providing a rich and emotionally colorful set of insights.

Here’s a look at how we set the stage for success by tuning the study design and triggers for the unique attributes of mobile video surveys.

The unique nature of mobile video studies

We’ve found that in mobile video surveys, respondents are remarkably open, honest, and candid in their replies. This may be because mobile phones – given how much they are used – serve as virtual extensions of the human hand. The device becomes quite personal and intimate in that respect. Additionally, we often use mobile devices to communicate in confidence with people we trust. So in a sense, the technology, by association, takes on attributes of a confidante: people are comfortable sharing openly with it.

Furthermore, as a market research methodology, mobile video surveys are highly effective for anything to do with the home. Participants often record their responses while at home, where they typically feel comfortable being themselves and where they are surrounded by belongings and items they use every day.

The compounded effect of using a familiar, personal technology in a familiar, personal environment makes the results of mobile video research studies highly insightful.

Applying ethnography best practices to mobile video study design

New technologies and media have opened up new ways to apply ethnography best practices with increasing sophistication and excellent results. The market research industry, driven to stay ahead of consumer preferences and trends, has rightly seen mobile video surveys as a way to overcome some limitations of traditional research approaches.

Take focus groups, for example. The sheer power of close observation, a highly insightful research practice, is largely absent from focus group research. Focus group observers are often situated such that it’s impossible to see the nuances of participants’ facial expressions from 10+ feet away. Depending on the seating arrangement, an observer might see some participants only in profile, making reading expression virtually impossible.

In contrast, a mobile video study is up close and personal, with the mobile device either held at arm’s length or in close proximity; so it offers a level of facial and body language observation that’s quite intimate, yet uncomplicated by interaction with an interviewer or other people.

Show & Tell is another ethnography methodology especially well suited for mobile video studies. Participants responding to Show & Tell prompts tend to be much more animated and articulate answering questions because they are either in the environment they are being questioned about or quite literally holding the object they are talking about.

Why did we survey only women?

While statistics show that men are a significant and growing audience for home cleaning products, we were experimenting with research design and especially curious about infusing a research study with the mother/daughter dynamic.

Mom knows that I take pride in the cleanliness of my home, and its organization, but it’s just never good enough for her.

~ Leslie Stone

UNORTHODOX STUDY DESIGN = POTENT RESULTS

We’ve worked as ethnographers in marketing, advertising, product design, strategy and consumer insights, and are still fascinated every day by the insights that emerge from the studies we do; from the standpoints of both study design and human thought and behavior. In this case, it was genuinely amazing to see study design, technology and human experience come together in such a powerful way.

Lesson #1: Engage emotion to reach deeper insights

STUDY DESIGN: Mommy and me; setting the stage

We decided to ask participating Millennial women to benchmark themselves against their mothers. “How do you compare to your mother?” From a study design standpoint—or from any standpoint, really—that’s a loaded question with some magical power:

  • It sets a very emotional tone, starting with the first question.
  • It provides a harder edge to the research findings because people are not just talking about themselves, they are talking about how they are different from their mothers.
  • With this in mind, in the first clip of the study we asked participants to show us a picture of their mother (or parents), to help bring the mother/daughter or parent/daughter relationship vividly to mind in the moment. Loading a research study with a design that gets to deeper emotional territory will almost always result in more meaningful insights and set the stage for that emotion to carry through the rest of the study.

Understanding emotion is invaluable for a number of reasons:

  • Despite what people SAY they will do, emotion often overrides logic
  • There are so many options for consumers that understanding the emotional drivers for brand preference can be a powerful asset.
  • While most categories—like home cleaning—have the potential to be mundane, we believe every category has emotion in it that can motivate consumers to buy specific products or brands that resonate with them.
  • The most successful brands and companies typically create powerful emotional connections with consumers.

CONSUMER INSIGHTS: They are their mother’s daughters

Oh mama! Whether study participants remained adherents of their mother’s cleaning practices and philosophies, or whether they were outright rejectors of their mother’s way of doing things, they all had powerful emotional connections to home cleaning.

For many of respondents, their relationship to home cleaning started with early childhood memories of cleaning routines, scents, brands and products, all intertwined with their relationship with Mom.

As you might expect, many of the women adopted a very different set of brands from their mother’s loyalties. Typically, the brands Millennial women related to more had a different mission, vision, or purpose. They weren’t as much brands that had been around for generations, but they were brands with original narratives and associations that aligned more meaningfully with Millennials’ desire for a greater purpose or mission.

I think my mother would probably say, as far as my home cleaning products and home cleaning style goes, I’m doing the best I can with two little ones under the age of four. And I would say she understands the type of products that I buy and why I buy them. Organic products are just not what Mom chooses.

~ Julie

Lesson #2: Help participants paint a complete picture in full color

STUDY DESIGN: Transition from Culture/Category to Brand questions

We typically sequence questions in a way that helps participants show us if/how their relationship with the broader research topic aligns with the brand landscape. To achieve this, we start with broader topics about the culture and category, then narrow the focus to more specific product areas in order to approach the topic from a cultural level where the richest—and often highest ROI—insights are found.

In this study, we sequenced questions to follow this arc:

  • Their relationship with home cleaning, relative to their mother’s
  • Their relationship with home cleaning brands, relative to their mother’s
  • Their relationship with home cleaning products, relative to their mother’s
  • Their relationship with home cleaning purchasing, relative to their mother’s

CONSUMER INSIGHTS: A kinder, gentler brand landscape

Since Millennials tended on the whole to move away from brands their mothers groomed their daughters on, this can have significant impact on new brands launching, as well as on existing brands either repositioning themselves or extending into new areas.

From a product perspective, Millennial women tended to relate best to what we would refer to as “gentler” cleaning products. They cited products that had a less harsh chemical footprint and were perceived as more environmentally friendly. They also lit up at the idea of convenience. Wet wipes, for instance, were one of the most commonly cited products they lived by because of the ease with which they could be used. (Incidentally, respondents did not seem to associate that extra convenience with an increased environmental cost of the throw-away plastic tubs).

My number one home cleaning product would be my Windex Touch-Ups. I love this product. What makes this different from what my parents used? It’s convenient, the ease of use and the design: it’s kind of high-tech, I think, compared to an all-in vinegar mix that’s maybe something my parents would’ve concocted

~ Pamela

Lesson #4: Seek to understand the entire journey

STUDY DESIGN: Show & Tell methodology for product context

What’s a day in the life of a product? For product context, we had these women show us where they stored their cleaning supplies in the home, talk about the range of products found there, and discuss how they typically used the assortment of products.

By designing studies to investigate what happens to products pre, during and post use, mobile video surveys can help identify new opportunities for product marketers and designers to innovate around that product, including “product as service,” an escalating trend across categories. This is especially valuable in the re-order world, where increasingly popular subscription models help increase customer loyalty and lifetime customer value.

CONSUMER INSIGHTS: The untold story = new brand & product opportunities

By having participants show us their cleaning supplies, we were able to understand more deeply not only their favorite product(s), but also the brand clusters and assortments they purchased in other categories. It also gave us insight into their unmet needs when we could see, for example, how disorganized some of the supply areas were; many participants were reluctant to even show those areas because they felt embarrassed by the mess.

Would these participants want a product or system that made it easier for them to create and maintain a supply area they wouldn’t hesitate to show us or Mom? Absolutely.

I think my mom and my parents would say that I do keep my house in good clean state, but she would definitely be appalled by the number of different products that I have. She would say, it’s way too much and I should just stick to the old fashioned water and baking soda.

~ Grace

Lesson #5: Use mobile for an intimate, very human point of view

STUDY DESIGN: Get into their personal space, literally and emotionally

Seeing into people’s natural environments using mobile video surveys is a unique way to truly understand their world from their point of view—not just product or product context, but what else matters to them and the kinds of challenges they face every day.

Additionally, not only does the research methodology address the core research objective, it’s also valuable and easy to share throughout an organization, both upstream (into Product Design & Development, or even R&D) or downstream (into retail presentation and online buying).

So the Swiffer Dust and Shine with Febreze Lavender and Vanilla is my favorite cleaning product. […] It definitely differs from my parents, because they never had these things back in the day growing up, when cleaning, I think things were much more old-school.

~ Shaun

CONSUMER INSIGHTS: Under the sink can be as emotional as in their closet

Having consumers show you spaces in their homes that very few people ever see—like under kitchen counters and in laundry rooms—opens them up to share unexpected stories and details.It also opens the eyes of the researcher to the reality of these spaces.

Hearing in someone’s voice the pride about how she organized an area or seeing the anxiety on another’s face as she introduces a space in her home that’s chronically disorganized reveals all sorts of explicit and implicit insights.

EFFECTIVE USE OF MOBILE VIDEO STUDIES

Would participants have been so candid in a focus group or even in a filmed one-on-one, ethnographic-style interview? Unlikely. Certainly not without considerably more time invested in building rapport and trust.

In this study, each person’s familiarity with—and trust of—her mobile device led to surprisingly candid sharing. There was no unconscious bias, and no group think.

Specifically, using mobile video surveys in this study helped us:

  • Understand the prevailing attitudes, practices, brand affinities, and purchase habits of Millennial women as it related to home cleaning and home cleaning products;
  • Infuse the study with emotion beyond simply asking women about their habits and preferences;Identify clear attitudinal, behavioral, and emotional differences between Millennial women and their mothers;
  • Surface the broader shifts amongst Millennials and the home cleaning category towards simpler, less harsh, and more convenient products;
  • Identify all sorts of opportunities for new brands to disrupt the category and/or for existing brands to re-tool, re-position, or extend their brands;
  • Explore zones of innovation in which both new and established brands can identify opportunities for new services and experiences on top of the existing product portfolio, such as organizing ecosystems for storage, re-supply purchasing, and line extensions into tangential categories

ACKNOWLEDGMENTS

Special thanks to the people who shared their personal stories and insights with us as part of this mindswarms study.

How Mobile Video Surveys Helped DINE Launch a New Product - Fabric
When food innovation company DINE needed to make a snap decision about whether to introduce a new product to consumers, they turned to mindswarms. The feedback they...
Millennials & Home Cleaning - Fabric
In a recent study, mindswarms set out to understand unique generational considerations in how Millennial women relate to home cleaning, home cleaning brands, and home cleaning product...
Amplify Your Pitch with Mobile Video Surveys - Fabric
4 days away from a major pitch, BBDO knew that having authentic & timely consumer video insights would amplify their pitch. The result: they won.
How to Seal the Deal with Mobile Video Surveys - Fabric
What happens when you need consumer insights within a matter of days? Check out how Fabric helped a leading agency craft video surveys to win new business.
mindswarms Partners with Carnegie Mellon HCI Masters Class - Fabric
In a pilot study at Carnegie Mellon University, students enrolled in Human Computing Interaction Masters program used mindswarms video mobile surveys to get feedback on their idea...
Mobile Video Ethnography Dives Into the Teenage Mind - Fabric
What shapes the identities of teens today? California ad agency BSSP wanted to reach beyond data-driven assumptions to find out, going deep into the minds of Generation...
How micro moments are reshaping the travel customer journey - Fabric
There’s a lot of planning that goes into traveling. And people increasingly turn to devices for help. Here, Think with Google (powered by mindswarms) reveals the consumer...
Mobile Video Surveys Reframe Global Qual - Fabric
Rob Klingensmith, Chief Strategic Officer of Marcel Worldwide, worked with mindswarms to collect consumer insights in 5 different countries for Luxottica, makers of the world’s iconic Ray-Ban...
How mobile has redefined the consumer decision journey - Fabric
The consumer journey, from I-need-some-ideas through I-want-to-buy-it, is increasingly guided by the smartphone.This means retailers need to show up in a useful way online in order to...
Dr. Lautman - Professor at Wharton’s MBA program - on Mobile Video Surveys - Fabric
Wharton MBA adjunct professor Dr. Martin Lautman appreciates the benefits of big data. But when it comes to diving deeper in order to understand consumer emotion and...

case-studies-amplify-your-pitch

Amplify Your Pitch with Mobile Video Surveys

Four days away from a pitch to a major national bank, BBDO wanted to gauge consumer impressions and attitudes about banking, with an emphasis on the client’s brand. They knew that having authentic and timely consumer video insights to share with the client would amplify their pitch’s power. The result: they won.

The Situation

The call comes in from the pitch consultant with multiple agencies on the long list. Everyone is excited but, at the same time, you realize you’re up against some serious contenders and statistically you have a 1 in X chance of making the final round. You need to get the biz dev wheels turning fast. You’d love some qualitative research to have a POV to start informing the creative brief, but OOP expenses are strictly managed by the CFO and every group is asking for funding. Focus groups are seen to be too traditional or you can’t afford a national sample; plus, time doesn’t permit because you have one week to get the next round of pitch material together.

Perfect occasion for Fabric; perfect occasion for high quality mobile video surveys.

The Solution

A 15-person Fabric survey. Bingo. 7 questions per person gives you over 90 minutes of video responses (respondents each have 1 minute to respond to each question). You have a national or global sample of people answering very specific questions about the client’s brand or product that you can bake into a presentation or even edit into a quick video. Heck, create a pitch manifesto for your approach based on the insights gathered, and leapfrog those other agencies. Responses “ship as they fill” so you and everyone on the pitch team reviewing responses as they come in. You can start to inform your internal teams about the consumer POV and use it to influence the way they think about creative, media, and every other aspect of the pitch.

I walked into the agency at 9:00am on Monday morning with a deck full of films, photos and verbatims. Heroic.

~ Gordon McLean, SVP Group Planning Director, BBDO NY

How They Did It

Working with Fabric mobile video surveys, BBDO quickly locked down the recruitment specs. They wanted a 20 person survey with an equal spread across gender, race, and geographic location, while also specifying other key needs; brand awareness, and regular use of the brand’s banking services. Fabric immediately created a screener to ensure that candidates met all basic demographic, geographic, and psychographic requirements. Recruitment began hours later. Together, BBDO and Fabric developed the survey questionnaire for participants. Conducted asynchronously over mobile devices (or webcam) via Fabric, the survey began the same day BBDO reached out to Fabric. By Monday morning, the participants had completed the survey, netting BBDO one hundred forty minutes of consumer insight to pick and choose from to both inform and complement their pitch.

Outcome

BBDO was delivered a cloud based study link that allowed for download or online viewing of all video responses. Using that study link in the pitch, BBDO showed a board with twelve choice participants, including their photos and summaries of their responses. To accentuate specific points, BBDO played poignant video responses for the client. Mobile video surveys helped anchor BBDO’s creative strategy in genuine consumer insight (vs agency staff speculation), making the client much more receptive to their ideas. In the end, BBDO won the pitch.

How Mobile Video Surveys Helped DINE Launch a New Product - Fabric
When food innovation company DINE needed to make a snap decision about whether to introduce a new product to consumers, they turned to mindswarms. The feedback they...
Millennials & Home Cleaning - Fabric
In a recent study, mindswarms set out to understand unique generational considerations in how Millennial women relate to home cleaning, home cleaning brands, and home cleaning product...
Amplify Your Pitch with Mobile Video Surveys - Fabric
4 days away from a major pitch, BBDO knew that having authentic & timely consumer video insights would amplify their pitch. The result: they won.
How to Seal the Deal with Mobile Video Surveys - Fabric
What happens when you need consumer insights within a matter of days? Check out how Fabric helped a leading agency craft video surveys to win new business.
mindswarms Partners with Carnegie Mellon HCI Masters Class - Fabric
In a pilot study at Carnegie Mellon University, students enrolled in Human Computing Interaction Masters program used mindswarms video mobile surveys to get feedback on their idea...
Mobile Video Ethnography Dives Into the Teenage Mind - Fabric
What shapes the identities of teens today? California ad agency BSSP wanted to reach beyond data-driven assumptions to find out, going deep into the minds of Generation...
How micro moments are reshaping the travel customer journey - Fabric
There’s a lot of planning that goes into traveling. And people increasingly turn to devices for help. Here, Think with Google (powered by mindswarms) reveals the consumer...
Mobile Video Surveys Reframe Global Qual - Fabric
Rob Klingensmith, Chief Strategic Officer of Marcel Worldwide, worked with mindswarms to collect consumer insights in 5 different countries for Luxottica, makers of the world’s iconic Ray-Ban...
How mobile has redefined the consumer decision journey - Fabric
The consumer journey, from I-need-some-ideas through I-want-to-buy-it, is increasingly guided by the smartphone.This means retailers need to show up in a useful way online in order to...
Dr. Lautman - Professor at Wharton’s MBA program - on Mobile Video Surveys - Fabric
Wharton MBA adjunct professor Dr. Martin Lautman appreciates the benefits of big data. But when it comes to diving deeper in order to understand consumer emotion and...