The Promise of Deep Learning

The Promise of Deep Learning

By Yoshua Bengio

Humans have long dreamed of creating machines that think. More than 100 years before the first programmable computer was built, inventors wondered whether devices made of rods and gears might become intelligent. And when Alan Turing, one of the pioneers of computing in the 1940s, set a goal for computer science, he described a test, later dubbed the Turing Test, which measured a computer’s performance against the behavior of humans.

Read: Accelerating Cognitive Computing

In the early days of my academic field, artificial intelligence, scientists tackled problems that were difficult for humans but relatively easy for computers–such as large-scale mathematical calculations. In more recent years, we’re taking on tasks that are easy for people to perform but hard to describe to a machine–tasks humans solve “without thinking,” such as recognizing spoken words or faces in a crowd.

That more difficult quest gave rise to the domain of machine learning, the ability of machines to learn. This is what interests me. It’s not really my goal to make machines that think like humans do. My aim is to understand the fundamental principles that may enable an entity, machine or living being, to be intelligent. I have long ago made the bet that this would happen thanks to the ability of such an entity to learn, and my focus is on building machines that can learn and understand the world by themselves, i.e., learn to make sense of it.

The reason I’m laying out this chronology is that I believe we’re at a turning point in the history of artificial intelligence–and, indeed, computing itself. Thanks to more powerful computers, the availability of large and varied datasets, and advances in algorithms, we’re able to cross a threshold that has long held back computer science. Machine learning is shifting from a highly manual process where humans have had to design good representations for each task of interest into an automated process where machines learn more like babies do — through experience –  building internal representations that help to make sense of the world. This is the field of deep learning.

Deep learning isn’t brand new. Indeed, when I was a student in the 1980s, it was the concept of neural networks, the precursor of deep learning, that got me interested in pursuing an academic career in computer science. What’s new is that the accumulation of many scientific and technical advances has yielded breakthroughs in AI applications such as speech recognition, computer vision, and natural language processing.  This has brought into the field a large group of researchers, mostly graduate students, and we’re now making progress in deep learning at a gallop.

We’re able to do that because of advances in creating hierarchies of concepts and representations that computers discover by themselves. The hierarchies allow a computer to learn complicated concepts by building them out of simpler ones.  This is also how humans learn and build their understanding of the world; they gradually refine their model of the world to better fit what they observe and discover new ideas from the composition of older ones, new ideas that help them to better fit the evidence, the data.

For example, a deep learning system can represent the concept of an image of a cat by combining simpler concepts, such as corners and contours, which are in turn defined in terms of edges. But we don’t have to teach it explicitly about these intermediate concepts, it learns them on its own. We don’t have to show the system pictures of all the possible cat colors, shapes, and behaviors for such object recognition systems to correctly identify that it is a Siamese cat that’s somersaulting in a photograph. When it “sees” a cat, it “knows” it is one.

I’m privileged to be part of a troika of computer scientists who are widely credited with spearheading advances in this field–along with Geoffrey Hinton and Yann LeCun. We co-authored a paper, Deep Learning,which was published in the journal Nature in May, where we laid out the promise of our branch of A.I. But this isn’t a field where a few “media stars” are doing all that needs to be done. To produce the advances that are possible and to find applications for them will require thousands of scientists and engineers–in academia and in industry.

That’s why I’ve been dedicated to rallying people to our exciting project. I’m co-authoring a book, Deep Learning, with Ian Goodfellow and Aaron Courville. Our core audiences are university students studying machine learning and software engineers working in a wide variety of industries that are likely to find important uses for it. This book-in-progress is posted on the Web, and we welcome people to read, learn and give us feedback.

Which brings me to another key point: I’m an advocate of open science. Like open source developers, participants in the open science movement believe that we should share knowledge as soon as we gain it to increase the pace at which the boundaries of science are pushed, and for the benefit of all. Many of my research colleagues and I contribute all of our deep learning inventions to the Theano project and its derivatives on GitHub. There, anybody who is building deep learning systems can use the algorithms and programming tools, and we urge them to contribute back to the project: hundreds already do so.

Just as sharing is essential to open science, so is collaboration–the kind that’s done transparently. The whole enterprise of science is a giant brainstorm. The Montreal Institute for Learning Algorithms (MILA), with its 60 researchers — including 5 professors, contributes to it via numerous collaborative research projects with scientists in universities and industry.

The newest of our collaborative research partners is IBM. We look forward to working with scientists and engineers in IBM Research and the Watson Group on a very ambitious research agenda, including deep learning for language, speech and vision. We believe that, together, we’ll be able to scale up and extend deep learning methods by using powerful computers to take on very large datasets. It will help machines learn more, across broader domains, faster and from a larger set of data sources, including the vast amounts of unlabeled data – that have not been curated by humans.

I’m tremendously excited about the future of deep learning. We’ve made rapid progress, and while we’re far from solving the great riddle of what it will take to enable machines to truly understand the world, I’m very hopeful that we’ll crack it.

And then the floodgates will open. Once computers truly understand text, speech, images and sounds, they will become our indispensible assistants. This will revolutionize the way we interact with computers, helping us live more conveniently in our day-to-day lives and perform more effectively at work. It will enable society to take on some of the grand challenges that matter to us–such as curing deadly diseases and spreading knowledge and wealth more broadly. As importantly, it will help us understand who we are and that part of who we are that has always fascinated me, i.e., how intelligence arises. This has been my dream for more than 30 years, and it’s fast becoming our reality.