Imagine you are an unscrupulous Market Research* company who enjoys both Pepsi and Coke as clients. They both ask you independently to conduct a taste test** on consumers, each in the knowledge and expectation that their superior product will win. How can you run tests that will keep both of them happy?
It’s easy. Both tests are of course scrupulously fair, conducted blind, with a random sample of consumers. But for the Pepsi one you ask consumers to just take a sip of the product, and for the Coke one you ask consumers to drink a small glass of it. Pepsi is sweeter, and so the initial sip leaves consumers wanting more, and they mark the product higher. Whereas over an actual glass Coke tends to win. Both tests are valid – both clients are happy.
*No MR company would actually do this, as blind product tests are pretty useless really*** – this is just an example of an effect that you find testing colas, that we use at work as an example of why you need rigorous testing protocols.
**(This is a test for preference not for recognition. Tests for recognition don’t work, obviously, because people assume that the product they like best is the one they usually drink and then distort the scores further by marking it higher.)
*** I believe the comedian Mark Thomas has some material on cola testing but I haven’t seen it.