Managing Big Data

Artificial Intelligence will change the world in both subtle and overt ways. AI has been employed in smart devices, online search engines, self-driving cars, advertising, business operations, and military intelligence to name just a few. These myriad applications all have one thing in common and that is “big data”. Big data is defined as “extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions.”

To really understand and appreciate “big data”, imagine every email that you have ever sent, printed out and put in boxes. Depending on your email usage, this may be one box or a room full of boxes. Either way, that would be an example of a relatively small “big data” set. Your email service provider has that data, and without too much trouble they are using it to make many inferences and predictions about your behavior.

In the past, this type of analysis would have required many weeks or months of work for a team of analysts to sort through and draw conclusions, even if they were using computers. This is a problem that has been solved by Artificial Intelligence.

How Machine Learning and AI Improve Processes

To explore this example further, consider the follow. I may want my email service provider to make it easier for me to see and reply to emails that are important to me. I also want the service provider to do this without me explicitly describing what makes any particular email important.

A computer programmer given this task would have to get answers to questions like:

•         Is an email more important if it comes from a particular person
•         …if my own address is in the To line
•         …if my name is written in the first line of the body of the email?
•         Is an email less important if I’m just CC’ed?
•         Does the algorithm account for how quickly I open the email after receiving it or if it languishes in my inbox unopened?
•         Does the algorithm consider whether or not I provide a response?

There are hundreds or thousands of criteria a computer programmer would have to consider to provide an accurate categorization of the importance of each email I receive. Also, the weight or importance of each of those criteria may be different for every user. To provide a truly useful and accurate categorization algorithm, programmers would have to write a customized algorithm for each user; this is obviously impossible. Artificial Intelligence can analyze all this data and answer the above questions based on each user’s behavior.

AI’s Benefits and Disadvantages  

Email is just one example. The computer systems many of us interact with every moment of every day generate data about us: behavior, preferences, routines, habits, etc. All this data is cataloged by the companies that generate it and that data is then used to answer questions that those companies have. Perhaps,  how they can improve their product line or services, or the best methods to sell you more.

These are relatively harmless applications for this data but what if a company that sells products online has determined that you are more likely to pay more for a product right after payday. This information could be used to list higher prices for a product in the days right before and after payday. The data we generate can be used not just to enhance our user experience but also to manipulate our behavior like improving the odds that I will purchase a product or service, or to maximize the price or volume of that purchase.