Deep Learning - Part 1

What Deep Learning Can Do?

A little marketing video of Project Adam from MS has opened up my eyes of what deep learning can possibly do.

John Platt from MS told you that deep learning can address the problem of representation. The beauty of it is that you no longer needs to hand-engineered the features and the machine can do a better job than you automagically.

Trishul Chilimbi further describes the magic behind it. Deep learning is a way to connect different regular neural nets together and have each layer performs a level deeper of representation. It is additive and synergetic so what you have train in task 1 can be retained when you do task 2 and the new task can benefit the old tasks.

What Andrew Ng says about it?

In 2012, Andrew Ng had an intro of Deep Learning. Here are few takeaway from his 45 minutes lecture:

There is a theory that human intelligence stems from a single algorithm. The idea arises from experiments suggesting that the portion of your brain dedicated to processing sound from your ears could also handle sight for your eyes. This implies that our brain is a general-purpose machine that can be tuned to specific tasks.
Andrew also showed us that deep learning has now outpaced supervised learning in some of areas.
The next topics is how we can scale it up as the research shows us that more data and bigger model will have a bette result.

Reference

If you are fascinating by this "one algorithm" theory, you can read about it here from Jeff Hwakins
Google Brain

OK, What is Deep Learning after all?

Deep Learning is a first step in this new direction. Basically, it involves building neural networks — networks that mimic the behavior of the human brain. Much like the brain, these multi-layered computer networks can gather information and react to it. They can build up an understanding of what objects look or sound like.

How hot it is?

Yoshua Bengio, an AI researcher at the University of Montreal, estimates that there are only about 50 experts worldwide in deep learning, many of whom are still graduate students. He estimated that DeepMind employed about a dozen of them on its staff of about 50. “I think this is the main reason that Google bought DeepMind. It has one of the largest concentrations of deep learning experts,” Bengio says.

Run on GPU?

Many deep learning systems are now moving to GPUs as a way of avoiding communications bottlenecks, but the whole point of Adam, says Chilimbi, is that it takes a different route.

Follow Who?

Andrew Ng - Baidu
Yann LeCun and Rob Fergus - Facebook AI
Geoffrey Hinton - Google Brain
Peter Lee - Microsoft Research
Bengio