Quantitative research studies (e.g., surveys or medical trials) and qualitative research studies (e.g., interviews or focus groups) are powerful tools in the hands of a trained marketer. Both forms of research are valid, and both provide useful insight into a market space, when properly conducted and appropriately interpreted. Yet they are as different as oil and water.
Some of the differences between these processes are obvious, even at first glance. Each approach has its unique methodologies, its inherent advantages, and its limitations. And neither approach works in all situations. Sometimes, marketers will begin their research with exploratory interviews, to uncover issues and develop theories, and then test those theories with broad, quantitative studies. At other times, empirical research may lead to creative concepts that must be tested in qualitative studies, like focus groups. But what happens when quantitative and qualitative research projects run concurrently? Are there dangers in mixing these very different processes?
The short answer is “yes.”
Case Study
Consider the case of a service organization with five distinct stakeholder segments. Let’s call them customers, prospective customers, delivery partners, the general public, and internal managers. (The “internal managers group” does not refer to a specific group of people or a job function, but rather to all of the mid-level managers throughout the organization, who collectively control the content and quality of the services delivered.) Now imagine you’re the Marketing Director for that organization, and you’ve been given the opportunity to hold focus groups with representative members of these five market segments, to ensure the organization develops services that meet the needs of its markets. One of the segments, the internal managers – a highly splintered but politically influential group – holds the power to define and deliver the very services your organization offers. And more than any of the other groups, the internal managers have the ear of your executive leadership.
Concerned about keeping peace with the internal managers, the chief executive approaches you and asks that you give the internal managers a chance to voice their opinions in a broader survey. A survey methodology will allow all members of this large, vocal, and diverse group to express their individual opinions. Moreover, the survey responses will enable the organization’s leadership to “segment and size the internal manager market,” and discern which of the influential splinter groups are representative of the majority opinion. Finally, the executive suggests that the composite viewpoint of the internal manager group, as represented by quantitative analysis of the survey, be weighed equally against the other four segments’ opinions, as expressed in individual focus groups.
The executive’s approach sounds reasonable enough, and even respectful of the science of each methodology. Yet, it’s fraught with perils.
Fundamentally Different Processes
Focus groups are designed to provide a window of insight into issues that might exist more broadly in a population. Participants are encouraged to answer open-ended questions and engage in unfiltered dialog about their perspectives. When professionally facilitated, focus groups allow the moderator to probe more deeply into comments made by participants. This provides deeper insight into the feelings and reactions of individual participants. By contrast, quantitative studies (e.g., surveys) probe less deeply on individual issues, but provide empirical data, which can be projected, with some validity, from the sample space to the entire population.
Mixed Media, Mixed Messages
One of the dangers of running quantitative and qualitative research concurrently, is the potential to confuse the processes. While artists may choose to mix techniques (as a watercolorist would mix wet-on-dry and wet-on-wet techniques) to create a beautiful and compelling piece of art, researchers who run concurrent qualitative and quantitative studies often end up with quite a mess. Readers of mixed studies, being influenced by projections in the quantitative portion of the study, may try to extend observations from the qualitative portion of the study, and end up with invalid conclusions. In your case, presenting the conclusions of your qualitative focus group research in the same context as your quantitative internal-manager-group survey results may encourage those who review the findings to project the opinions expressed in the qualitative studies (focus groups) statistically, to the larger population, and this would be inappropriate. The resulting, inaccurate view of your market space will skew the development of your services matrix.
Non-Comparable Weighting
People who look at qualitative and quantitative studies side-by-side tend to assign more weight to the statistical (quantitative) results.
This is particularly true in scientific communities (where experimentation and research define the discipline), and in manufacturing (where continuous improvement is defined in terms of deviation from specifications), and in highly technical environments (like IT), and in business environments (where well quantified ideas are more likely to be funded).
In your case, the stated purpose of the research was “to ensure the organization develops services that meet the needs of its markets.” And eventually, these developments will need to be funded. So, when it comes time for you to present your statistical and qualitative findings simultaneously, your organization’s decision-makers may place undue emphasis on the statistical results, and could even dismiss the findings of the qualitative research.
Assumptions of Relative Homogeneity or Diversity
When we first presented the case study example, we defined the internal managers group as “a highly splintered but politically influential group.” The splintered nature of the group was also a concern of the chief executive, who hoped that a broad survey of the internal manager population would help “discern which of the influential splinter groups are representative of the majority opinion.”
But the decision to approach the internal managers group with a different methodology based on the fact that they are diverse inherently implies that the other groups are each homogeneous. And that is a dangerous assumption.
In fact, it is likely that one or more of the other stakeholder groups is as diverse and segmentable as the internal manager group. But the decision to employ a statistical survey with the internal managers group (and not with the remaining groups) will bias you toward identifying and defining the subgroups of internal managers, while treating the remaining groups as individual but homogenous groups who simply happen to share concerns.
Furthermore, you could easily end up comparing not five, but six or seven or eight groups, several of which would be subgroups of the internal managers group. This would give undue influence to the internal managers segment in the final decision-making process.
Relative Importance of One Group Over Another
Ultimately, as marketing director in this fictitious organization, you will need to recommend a course of action. That course of action should give relative weight to the needs and opinions of the groups based on their strategic priority to the organization. Working closely with your CEO and applying the corporate vision to your thinking, you may decide that one group – for example, “existing customers” – should be the focus of your organization’s energies. But when you do set priorities, you will need to anticipate a response from the “highly splintered but politically influential group [that] holds the power to define and deliver the very services your organization offers.” And because you have allowed that group to have its own empirical study, that group will be armed with statistics to reinforce their position.
Show Me a Way Out!
All of these concerns can be abated (or at least minimized) by conducting the quantitative survey after the focus groups are concluded. In this two-tiered approach, your first study (focus groups) would provide your organization with comparable qualitative feedback from all five primary stakeholder groups, and the second study (surveys) would give your CEO the insight he needs to understand how large a percentage of the internal managers population holds each of the conflicting viewpoints.
What’s more, your organization will find three additional benefits to the two-tiered approach:
- By running the focus groups before the survey study, you will have the opportunity to identify (narrow down) the most important issues and discuss their weighting with your CEO. This will help you phrase the questions for the empirical research, and provide a sense of “focus.”
- Further, the questions you develop for the survey could be equally applicable to several audiences. So, rather than gathering statistical feedback from internal managers only, you could simultaneously target multiple audience groups, including customers, delivery partners, the general public, etc.
- Then, by comparing the survey responses from each group, you could clearly see the differences in values and needs among your target audiences. This would yield valuable “needs profiles” to which you could map message platforms.
By separating the surveys from the focus groups, you maintain the integrity of each process, deliver clearer and more meaningful results, and enhance the ROI of your survey. In the end, your organization develops a more accurate understanding of its markets, your CEO gains valuable insight into his manager-constituents, and you get to retain your sanity.
See? It’s good to be the Marketing Director!
