Deep Learning

with a bit of Javascript

Martin Andrews / @redcatlabs

18 December 2014

Motivation

  • My focus : Machine Learning
  • Recent presentation from Google covered a lot :
  • There's a good JS implementation :

Current Applications

  • "Deep Learning" is a hot topic
  • Not just research : currently being used

House Numbers

Google Street-View (and ReCaptchas)

House Numbers

Speech Recognition

In Android phones now (since Jellybean, 2012)

Speech Recognition

Trained in <5 days on 800 machine cluster

Auto-Labelling

Part of Google+

Auto-Label

Basic Approach

  • Same as original Neural Networks in 1980s/1990s
  • Simple mathematical units combine to compute a complex function
Image to Cat

Supervised; Unsupervised; and Reinforcement Learning

Single "Neuron"

Different weights compute different functions

One Neuron

Multi-Layer

Layers of neurons combine and
can form more complex functions

Multi-Layer

Supervised Learning

  • while not done :
    • Pick a random training case (xtarget_y)
    • Evaluate output_y from the x
    • Modify the weights so that output_y is closer to target_y for that x

Gradient Descent

Follow the gradient of the error
w.r.t the connection weights

Gradient-Descent

10 Step Program

Real neurons aren't particularly fast...

10 Steps max

ConvnetJS

  • Clean code on karpathy's GitHub
  • And an excellent demo site :
    • Sample LIVE demos
    • Background Reading
    • Documentation
  • MNIST description (and 10secs of demo!)

Some more Details

Convolution

Other 'tricks'

  1. Use LOTS of real data
  2. 'Create' data :
    • Change the data in ways that have no visual effect
    • Ask the Internet to label images for you
  3. Clusters, GPUs, clusters-of-GPUs

ConvnetJS efficiency

  • Matrix multiplies are a big part of ConvNets
  • Straight JS does more house-keeping than necessary
  • Possible to optimize in several ways

Precompute stuff

Avoid unnecessary object manipulation


- if(oy>=0 && oy<V.sy && ox>=0 && ox<V.sx) {
  
+ if(oy>=0 && oy<V_sy && ox>=0 && ox<V_sx) {  
   

Type Hinting

This is an idea from asm.js


- var A = new Vol(this.out_sx, this.out_sy, this.out_depth, 0.0);

+ var A = new Vol(this.out_sx |0, this.out_sy |0, this.out_depth |0, 0.0);
+
+ var V_sx = V.sx |0;
+ var V_sy = V.sy |0;
+ var xy_stride = this.stride |0;  
   

Index Shuffling

  • Get the data to fit into on-chip caches
  • And move code out of loops if possible

- for(var fx=0;fx<f.sx;fx++) {
-   for(var fy=0;fy<f.sy;fy++) {
-     for(var fd=0;fd<f.depth;fd++) {
-       var oy = y+fy; // coordinates in the original input array coordinates
-       var ox = x+fx;

+ for(var fy=0;fy<f.sy;fy++) {
+   var oy = y+fy; // coordinates in the original input array coordinates
+   for(var fx=0;fx<f.sx;fx++) {
+     var ox = x+fx;
+     for(var fd=0;fd<f.depth;fd++) {
  
        a += f.w[((f.sx * fy)+fx)*f.depth+fd] * V.w[((V_sx * oy)+ox)*V.depth+fd];
   

Net Result

  • Speed-up Factor : 2x
  • NB: No change in functionality

Back to Demo

  • The demo site should now show :
    • How it actually learns
    • Results of the training
    • How the net is actually 'deeper'

The Frontier

  • ImageNet Competition
  • Natural Language Processing : Word Embedding
  • Other stuff :
    • Reinforcement Learning (DeepMind ~ Atari)
    • Images and Words

ImageNet Competition

Typical images

ImageNet

Deep ConvNet

  • Google won 2014 ImageNet
  • 24 layer deep network
  • 6.66% Top-5 error rate
Google ImageNet

Good Classification

ImageNet Flowers

"hibiscus" vs. "dahlia"

Good Generalization

Both recognized as 'Meal'

ImageNet Meal

Surprising

Image Search Yoda

Word Embedding

How to handle 'discrete' things like Words?

Word Embedding

Trainable on Wikipedia

Word Embedding Training

5.7MM documents, 5.4Bn terms
→ 155k words, 500-D embedding

Finds Relationships

Word Embedding Facts

Semantic too

Word Embedding Grammar

Wrap-up

  • Deep Learning is 'hot' for a reason
  • Main difference from the past :
    • Quantity of Data available
    • Amount of Compute power available
  • Have fun with ConvnetJS ...

- QUESTIONS -


Martin.Andrews @
RedCatLabs.com


( 'mdda' on GitHub! )