Using Qualitative Data
Sometimes I conduct focus groups or a set of interviews for an organization whose decision makers want to use the findings right away. They need the data to develop policy or improve programs, products or systems that need fixing sooner rather than later. These clients know their industry, understand the value of hearing directly from their constituents or staff, and have some understanding of the issues they want to address. They don’t need a lengthy report with background, literature search, methodology, and conclusions.
What they need instead is someone to collect the data in a rigorous and thorough manner and organize the findings in a format they can easily access. I like these kind of “hit-the-ground-running” projects because they are so utilization focused. The questions are generally pragmatic, intentional, and likely to generate useful information. And, rather than wasting away in a thick report that sits for months on decision-makers’ bed stands, findings are generally put to use within a month or two after they are generated.
So, for this type of client project I organize the data I have collected in a multiple page Excel spreadsheet in place of a report. On the spreadsheet, each response is entered on a separate line, coded by category and organized by question. Each response is also accompanied by face codes—gender, race/ethnicity, age, longevity in the profession/work place, and title/position—that may be helpful for further data sorts by subgroup.
But the hand-off doesn’t end there. Because qualitative results require a bit more skill to interpret and apply than quantitative data, I like to spend time talking with the client about how to get the most out of the data and use it in an objective, thoughtful way.
Over the years I’ve developed guidelines that I encourage clients to consider as they sift through the spreadsheets. Here are eight of the ones I never fail to mention:
- Be careful not to over-interpret the data. Don’t read more into the data than can actually be substantiated by the findings. It’s a natural tendency to “fill in the blanks” with data from our own mental files. Instead, be curious about options other than the ones you’re most familiar with.
- Resist the urge to quantify qualitative data. Because many of us are more comfortable making decisions based on numbers we have a tendency to count up the number of times a particular comment was made. Qualitative data can “point” us to several areas that need addressing but cannot quantify the extent to which any of those areas is an issue in the larger population. It’s not that similar frequently occurring comments don’t indicate an area to which we should pay particular attention. It’s just that open-ended responses collected from a small number of participants not randomly selected may not be statistically significant. Actually, an issue brought up by two people can be bigger than one brought up by seven or twenty people if the two people have an in-depth understanding of the issue and speak for several others not included in the group. When it’s important to quantify qualitative findings, follow up with a survey distributed to the bigger population from which focus group or interview participants were drawn rather than counting up qualitative responses. The two approaches are very different and do not share the same analysis logic. Counting qualitative data could easily lead to the wrong conclusion.
- Use your whole brain to interpret the findings. Interpretation is tricky work. We need to capitalize on the logical, analytical, objective left brain while at the same time involving the intuitive, thoughtful and subjective right brain. If we only needed the logical left brain, we could have a computer do the analysis for us. But that’s just not the case. Interpretation of qualitative data depends on a combined creative-rational process that integrates the best of both sides of our brain. So let your left brain consider the frequency of similar comments, the best categories for organizing the information and if certain comments make sense in light of other comments. At the same time, allow your right brain the freedom to integrate information from outside sources, draw on previous experience with similar topics and detect emerging patterns and themes.
- Don’t get hung up on any one response. Qualitative research is not investigative reporting. We’re not looking for one or two key clues that lead to a complete understanding of the issues. Instead, try to capitalize on the collective wisdom of the group of interviewees or focus group participants.
- Look for convergence of data from different sources. This occurs when multiple respondents provide a similar answer to any one question, when respondents across diverse roles or age categories say the same things, or when responses to different questions all point to the same conclusion. These points of convergence signify major themes in the data. This is where the power lies in qualitative data. Pay particular attention to what these convergences tell you about the population you seek to serve.
- Pay attention to saturation, but not too much. When no new information is revealed about a topic with each subsequent focus group or interview it generally means respondents have reached a point of saturation. In other words, there isn’t necessarily anything else new to say about the topic. Saturation, like convergence, adds power to the finding. But be careful not to fall into the counting trap. Just because the same point is made 38 times (33 times more than it needs to be made) doesn’t mean it is any more relevant to the issue than a point that hasn’t reached saturation.
- Resist the tendency to identify the individual behind any one response. Particularly in small organizations and rural communities, it can be easy to identify some individuals by the tenor of their response or certain words they always use. Set aside what you know, or think you know, about who said what so you can look at that comment as objectively as you do all of the other responses.
- Contextualize the data. Consider qualitative findings within the context of other available information. This may include other data sets (survey results, etc.), past history in addressing the problem (successful or otherwise), constituent demographics, organizational politics, budgetary realities and internal capacity for addressing the issue. What the data means or how it gets used can change dramatically depending on any of these contextual variables.
Nothing excites me more than hearing about how people “take the big leap” to use the qualitative data they collect to make changes in their organizations, programs and products. Based on your experiences, do you have other suggestions to add to the list?
Leave a comment below and let me know if you’d like me to expand on any of these in a future blog.
Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.