Human Intuition to be replaced by the Data Science Machine

Big-data analysis is all about looking for patterns that are suppressed but, have some predictive power. But for this some features of data have to be selected to be examined and this of course requires human intuition. For example, in database encompassing consisting of times periods of sales promotions and profits, the dates only might contain the vital information, but the time span between them or average of those time spans. Researchers at MIT are working on removing this human intuition requirement in big-data analysis. They think of doing this with the help of an innovative method that will search the patterns and will also design the feature set. The first prototype of the system is ready and was registered with 3 data science competitions. In the competition the system competed against human teams to see who can better detect the predictive patterns in the given data. Around 906 teams participated in the competition and the “Data Science Machine” beat 615 teams.

In two of the competitions, the predictions made by the Data Science Machine came to be 96 percent and 94 percent correct and made it to winning submissions. However, in the third competition the result was at 87 percent. But the surprising thing to be noted is that the Data Science Machine took only two to twelve hours for analyzing the entries whereas the human teams took months to develop their prediction algorithms.

Max Kanter’s thesis in computer science has laid the foundation for this machine and states the researchers take this machine as a match to the human intelligence and there is lot of data to be analyzed. Kalyan Veeramachaneni, research scientist at MIT has worked on the thesis with Kanter. The paper will be presented at IEEE’s International Conference on Data Science and Advanced Analytics.

Veeramachaneni is the co-leader at Anyscale Learning for All group, CSAIL where the machine learning techniques are applied to practical problems related to big-data analysis. He said “What the researchers have seen from their experience in resolving a number of data science problems for industry is that one of the very acute step is named feature engineering. The initial thing you have to do is recognize what variables to bring out from the database or compose, and for that, you need to come up with a lot of ideas.”

Image Source: thestandarddaily.com