Numbers lie or when quant is powerless

Attitudes towards ad testing among marketers differ drastically: from complete rejection and distrust to a willingness to test everything, always, and in a variety of ways. In reality, there are different situations in marketing practise: sometimes testing can be skipped entirely, sometimes qualitative or quantitative research alone is enough, and sometimes both approaches are necessary to make the most effective decision

I want to share with you one of the examples from my experience that vividly illustrates this topic. And if you would like to learn more, welcome to the Insight Home for a cup of coffee (or let us meet online if you are not in Ukraine), and I will tell you a lot of useful things that I don’t write about in open sources

---

Once upon a time, when our agency "Mustang Research & Consulting" consisted of two research departments - qualitative and quantitative, we tested communication concepts for one beer mix brand

The classical approach involved a combination of qualitative and quantitative stages, where qual goes first, concepts are refined based on its results, and then taken to the quant

However, in our project, the qualitative and quantitative stages were carried out simultaneously. I don't remember why, but most likely, the client tried to save time, and, as it always occurs, in vain

According to the results of the qualitative stage, I saw the clear potential of one communication concept, while at the quantitative stage a completely different one "won"

My colleague, a quantitative researcher, and I began examining the data collected at both stages to figure out why the outcomes differed so significantly. And we quickly found the answer

The concept that "won" during quant by most key criteria was the simplest and easiest to understand for the target audience. Therefore, respondents reacted most positively, giving it high marks on the majority of questions, because when normal people appreciate something, they prefer to praise it without realizing that each question hides various criteria from a marketer's perspective. At the same time, the concept-leader from quant was category-generic and did not convey any added value of the Client’s brand

The communication concept that I recommended to the client based on the results of the qualitative stage was an outsider in quant because it included several messages at once, and people simply didn’t understand it. However, when I broke this idea into separate elements during the fieldwork and discussed them with respondents, it became clear that one of the "carefully hidden" messages had significant potential for the Client's brand. It motivated the target customers, was relevant to the brand, and advantageously differentiated it from competitors

Fortunately, we managed to convince the Client to choose the communication concept based on the qualitative research results, and it worked brilliantly. Moreover, this idea could have become an excellent foundation for the brand's positioning if the Client's team had not changed shortly thereafter, leading the brand in a completely different direction, as often happens. But that’s another story

---

What’s important to understand:

A skilled qual researcher can analyze respondents' perception of the material, starting from the overall concept and finishing with the effectiveness of the smallest details and their combinations. It is precisely the qualitative research method that gives the Client and their creative agency an understanding of what, why, and how to refine to achieve maximum effect

If several concepts show certain potential at the qualitative stage, then it makes sense to proceed to quant so to understand which idea is more effective. But only refined concepts should be tested quantitatively, because there is always the risk that the audience will choose something they like (for some unknown reason) rather than what will be effective for your business/brand

Our case clearly demonstrates that the qualitative stage should have preceded the quantitative one. Refined concepts, based on the principle of “one idea - one message,” would have ensured completely different results. And, most likely, the idea we recommended would have “won” in the quantitative stage as well

But honestly, the results of qual were enough, as at the level of identified cause-and-effect relationships and common sense, the potential of the idea was obvious. I know, I know, all the books say it differently, it's wrong, etc. But, as my clients often say: “I don’t care if it’s right or wrong, but it works. Thank you!” 😊

See you soon,

Your old-school qual researcher and the hostess of the Human & Sound Insight Home