Evolution of Analytics | Data Analytics

12 to 15 years back when I was working as a consultant in the pharma industry, the world was very different. We had the marketing team and the sales team and there used to be one analyst sitting in a corne who just provided us with the MIS (Management Information System) data, so we only had some data reporting and nothing else. We used to walk up to him and say I want to look at the sales performance of this brand in this territory broken up by some geographical details. Then he would need another three weeks or so to gather that data. So, anyone having such a requirement would walk up to the analyst and request him the data they need; and after all the hassle and hard work throughout the months, the analyst was unable to meet the various data needs of all these executives. You won't believe that I used to land up to this analyst every day and ask if my work has been done. At the end of all this, I used to receive a huge dot matrix printout, which I then had to convert into MS Excel, make a chart and use that data for all my consulting presentations.

The questions that I have answered as a consultant over the years are more or less the same, some of which are:

To answer any question, we used to make graphs; these were standard graphs like line charts, a bar charts, and a pie charts. Using these charts we used to build a presentation, give consultation and make money. That is the way it operated, and the data was very simple; just the sales data, a little bit of financial data and using these to get some data about the physicians from the field, that's all. To get such simple data, however, it took about three to four weeks, this is about 15 years back.

 

Then about 10 years back, things changed. I was in the USA, working with very large companies and that is when I moved into technology. At that point, the Oracle and other data warehouses were becoming very popular. They used to get all the data together and build a customised self-service reporting environment for the user. So all you need to do is, write a query, put the query in the database and you get a report. Voila! The report could be generated promptly but the data would take three weeks to refresh. The data comes in, gets cleaned and loaded and after that, you run a query and it's available. Meanwhile, if you want to change a parameter, that would take another three weeks. So you can imagine what we had to go through to get the report and get the information.

 

But the data was a challenge, and challenges cannot make the business wait. If your business wants a change, it will again take four weeks. If you look at the scenario from 5 years back (described earlier), the time taken to gather the data had not changed, it would still take three to four weeks to get the data you need. The only thing that changed was maybe the ease of getting the data. The queries were exactly the same, sales figures, geography, physician’s data and the reasons for growth or decline, and we made the same kind of charts like a bar chart or a line chart, or a pie chart into a beautiful presentation, gave it to our customers and made the money.

 

At this point, I realized that the data warehousing business is a pretty good business. During my sales meetings, I often used to say that I love the data warehousing business. Why? Because it takes about three years to build a data warehouse, and by the end of three years, it becomes redundant so you start all over again, which means you can get 15 million dollars worth of business every three years continuously for the next five years. That's the way the information management used to flow, and that was about ten years back.

 

Now move it to five years back. Five years ago, things changed, and even more so. Data channels increased, so we had the Multi-Channel Data; we have a lot of data coming from Insurance Companies, a lot of data coming from physicians themselves, and some bit from patients. Social media started blooming, so the data expanded, and thus, the technology also expanded. So we moved from the Oracle databases to Netezza and the likes. We could now handle data very easily. There were very nice platforms like Siberian and Relative, which were used to manage data quality, and everything improved, but the problem still remained the same. They were too many channels and we could not get it together; we had to integrate. How do you match these physicians, how do you match this persona? Again another four weeks to answer these questions. The question was exactly the same, and the charts also remained the same, and the money that we made was a little more, but everything was pretty much the same.

 

So if you look at these past 15 years, the questions have remained the same, and our representation remained the same; it is just that the funnel of data expanded exponentially, and that's why technology had to completely match with the business needs to make sure that it is able to handle the data.