Big Data: Is It Big Deal for Market Research Industry?

Posted in Operations & IT Articles, Total Reads: 2404 , Published on 02 August 2013

McKinsey, Forbes, Harvard and every organisation that matters has been keeping a tab of upcoming technologies and all enlist “Big Data” in their list. Does that mean big data is such a big deal that it can threaten or change the way market research industry operates? Let’s explore the scenario by looking at the definition of both Market Research and Big Data. This would help us understand the realm and objective of both.

Market Research is the systematic and objective identification, collection, analysis, dissemination and use of information for the purpose of improving decision making related to the identification and solution of the problems and opportunities in marketing.

Image Courtesy:

"Big data" refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. There has been a growing consensus in the industry to define big data, by three V's-

  • Volume - the magnitude of data has to be large, in petabytes not just gigabytes
  • Velocity - the data has to be frequent, daily or even real-time
  • Variety - the data is typically (but not always) unstructured (like videos, tweets, chats)

Now we can dig a bit deeper and see some points of parity and some points of disparity for ourselves.

Points of parity:

  • Purpose of both is to improve our decision making ability.
  • The mechanism is also similar to the extent but depend on data to improve our understanding of the problem at hand.

Points of disparity:

  • Big Data deals with extremely large datasets (in petabytes or more).
  • Big Data deals with mostly unstructured form of data and vice-versa.
  • Big Data problems have very short decesion making time. Following example from Forbes illustrates the point:"When you walk through the airport and they take pictures of everybody in the security line to match every face through facial recognition, they have to do that almost in real-time. That becomes a big data problem. If I am a bank and looking at a vast number of credit scores and histories, and I don’t need to provide an answer in five seconds but can do it next day, then that is not a big data problem."

So now with these similarities and differences in mind we can better answer our question.

Big data is indeed helpful in charting the domain not much covered by marketing research: “Unstructured social data”. The beauty of machine learning techniques and big data is the ability to learn pattern and this increases our knowledge base and helps us solve our marketing problems. There is a lot of social media data, telephonic/electronic feedback which can be data mined to obtain meaningful patterns. If we see this point in context of our discussion it means big data replacing use of some market research techniques such as panels and focus groups. Example: A car company wants to ascertain when people buy cars, let’s say, during festive season or appraisal time. We could check tweets and facebook posts to analyse context and time of the year where sales are more. The sales and CRM data could be used for cross verification.

So we see the merit in big data; it being able to replace use of panels and focus groups. Still it isn’t a silver bullet. Market research deals with data collection, qualitative and quantitative analysis and definitely big Data could certainly help in some of these areas, but not all. It is especially difficult to replace techniques like “Experimentation” and “Observation”. Big data also doesn’t help you with a problem where context or information is relatively less. Example: A cola giant wants to introduce a new flavour and wants to try out before launching it full-fledged. Here, we cannot apply big data analysis. Some people might say, “We could come to a conclusion that people may like vanilla cola by analysing their social data on foursquare, zomato, and facebook etc.” but nonetheless it’s not sufficient for our case. We would definitely like to experiment with the flavour, launch in a test market to know consumer reaction and this indeed is what market research does.

We can thus say there are occasions where we can embrace big data and leave out old market research techniques, but nonetheless we cannot abandon market research completely. The relation between big data and market research has some common intersection, but for other only one of them suffices.

Following example illustrates cases where only one of them seems appropriate:

Example 1: A SME wants to ascertain reasons for its falling sales.

Here using big data is not possible as we have limited datapoints, and also dealing with the complexity of the problem it would be like killing an ant with a nuclear weapon. In this case, market research helps us understand and solve the problem.

Example 2: Can you predict which state might be hit by cold next? If possible this would help chemists and stockist to prepare stores for appropriate medicines beforehand.

This is actually being done by Drug maker SSP Co. in Japan by analysing tweets. This problem is outside realm of market research, it needs dynamic datasets and powerful and fast computing, i.e. Big Data problem.

Example 3: Target group used their data on shopping patterns to predict if a teen was pregnant and all this with help of big data.

Considering above points we can say that big data is not a substitute for market research, but a helping hand. At best it could be said they have their own areas of problem solving with some overlap. Big data lets you uncover “what” part of the problem, but some times it’s more important to know “why”. Primary market research helps you answer “why” behind the “what”. The two approaches are scientific in nature and at best complimentary, but not a threat to each other.

This article has been authored by Manish Arora from DoMS IIT Roorkee


Marketing Research: An applied orientation 6th edition


If you are interested in writing articles for us, Submit Here