Over the past few months, the field of artificial intelligence has been exploding. A lot of people I meet here in the bay area talk about it constantly, and they try to come up with different use cases for artificial intelligence. It is increasingly clear that artificial intelligence will be a major toolset of the future. I believe it will exceed the status of a toolset and find an evolutionary path of its own.
But the more conversations I have around this, the more definitions I hear around the different buzzwords. What is artificial intelligence? Is it the same as machine learning? Some people throw around words like Natural Language Processing (NLP). What is that? Most predictive analytics companies claim to be using some form of artificial intelligence. Are they really all using cutting-edge technologies? If not, what are they using? And how does it help or hurt them when competing with other companies who, in fact, are using some of the cutting edge tools?
Over the series of the next few blog posts, we plan to illuminate the key differences between what people are doing, how to think about machine learning and AI in your product, and how to prepare your company to be competitive in the future that is inevitable.
But first, some definitions. Keep in mind that these blog posts are written from the point of view of practitioners and not researchers (although we work hand in glove with researchers). Thus, we won’t get super technical about any of these items. There are people far smarter and far more articulate who have done an excellent job of demystifying the science behind all of these concepts. We will do a blog post compiling some of our favorite resources very soon. For now, we will focus on the practical aspects of the field and how company executives should be thinking about the best ways to use data to put their companies on the far end of the competitive spectrum.
Ok, enough chatter. On to some loosey-goosey definitions, along with a recap of some of the basics:
What is big data?
So, you have heard the term big data and understand that it is a large amount of data that could be structured or unstructured. As you know, it is important because there are meanings, patterns and predictive behavior hidden in the large swath of data. However, traditional computational and data processing techniques that we all grew up studying just don’t solve the problem of understanding the meaning behind such large amounts of data. Firstly, this large amount of data needs to be stored across hundreds (or thousands) or servers. Then, it has to be presented in a format where the data can be analyzed. Traditional techniques of analyzing massive amounts of data in one go just don’t work. This is the main problem that traditional analysts have. They just can’t hold and analyze like they did in the past. Along with the proliferation of the cloud, newer big data techniques can help wrangle this large amount of data much more easily. This makes it easier to handle ‘big data’. Which brings us to the next question:
How do we make sense of all this data?
To make sense of the data, we first have to present it in a format that any algorithm can consume. The next part is tweaking those algorithms to get a desired understanding. Machine learning is one of the newer techniques that can help understand the patterns in the data without an analyst starting from a specific viewpoint. Actually, machine learning techniques have been around for decades (yes, decades). But in 2012, there was a major breakthrough that was able to get a phenomenal result in identifying handwritten digits. The technique that the researchers used came to be known as deep learning. Researchers, and then practitioners, all over the world rejoiced, and felt that this was the new silver bullet to solve the world’s data analysis problems. Coupled with the fact that everyone was generating vast amounts of data, researchers felt more confident that this technique + big data could find hidden meanings which were more difficult to find in the decades past. It looks like their excitement was well placed. Great progress has been done in this area, and the progress continues to surprise even the most ardent fans of the techniques.
So, machine learning lets computers find meanings in data?
In short, yes. But that’s a very broad definition. More specifically, machine learning refers to the idea of letting these new algorithms and techniques find meaning in data without starting from an analyst’s viewpoint. Let me give you an example. With data analysis, a typical analyst will come up with theories on how the data could be related and then validate those theories. Most of the time, their hypothesis proves incorrect, but not without giving them more information so that they can come up with a new hypothesis. Machine learning techniques turn this approach over on its head. By letting machines discover patterns in the data, they can be used to find highly complex relationships within the data which cannot be adequately modeled by the best of mathematicians. Exactly how they do this is the subject of another blog post, where we will cover basic concepts like supervised learning and unsupervised learning, and when each one makes sense. For now, let’s keep in mind that the machine learning techniques are more powerful and try to uncover patterns which the machine learning theorist or practitioner need not be aware of before the process begins.
Ok, I get it. Can machine learning be applied to ‘small data’?
Yes. It is not necessary that a large amount of data be present for the techniques to be successful. The simple way to think is whether the data contains enough information and structure to make some sense. For example, a list of 100 houses in a zipcode with prices and square footage will give one a very good idea how to price a new house given it’s square footage. However, if the data only contained house prices and the number of windows in the house, then that’s not a good indicator. The best way to think is that if a human can be trained to make some sense of the data without relying on other knowledge, then a machine can probably do so as well.
So, what is this artificial intelligence?Artificial intelligence is the most difficult one to define. I tried to read the definition on Wikipedia, and it gave me a headache. Everyone defines it differently, but in general it refers to the idea of computers and algorithms doing things that were earlier considered the dominion of humans. For example, understanding complex voice commands, sentences and phrases was considered near impossible about a decade ago, and yet, computers are able to do just that. Similarly, reading, characterizing and understanding handwritten signs, or the landscape while driving a car are all things that seem fantastic for a machine to be able to do. Ultimately, under the covers, it is a matter of getting a lot of information from various sources (multiple cameras and all kinds of sensors) and correlating it in a manner which is similar to how we make sense of the data. Hence, the term ‘artificial intelligence’ — there is a lot more complex “solving” and “learning” happening. Also, it sounds cool!
I hope the above gives you some sense of the world of machine learning and artificial intelligence. Over the next few posts, we will go a little deeper into each topic, while keeping in mind that our target audience are industry executives who should be prepared for the changes which are already occurring in their industries.
If you have any questions, feel free to email me or find me on linkedin. I’d love to hear from you if I can help you or your team with machine learning.
Information Overload 