Deep Learning Workshop

FOSSASIA 2016

Martin Andrews @ redcatlabs.com

19 March 2016

About Me

  • Machine Intelligence / Startups / Finance
    • Moved to Singapore in Sep-2013
  • 2014 = 'fun' :
    • Machine Learning, Deep Learning, NLP
    • Robots, drones
    • "MeetUp Pro"
  • Since 2015 = 'serious' :: NLP + deep learning

What can be done now

  • Speech recognition
  • Language translation
  • Vision :
    • Object recognition
    • Automatic captioning
  • Reinforcement Learning

Speech Recognition

Android feature since Jellybean (v4.3, 2012) using Cloud

Trained in ~5 days on 800 machine cluster

Speech Recognition

Embedded in phone since Android Lollipop (v5.0, 2014)

Translation

Google's Deep Models are on the phone

Google Translate

"Use your camera to translate text instantly in 26 languages"

Translations for typed text in 90 languages

House Numbers

Google Street-View (and ReCaptchas)

House Numbers

Better than human

ImageNet Results

ImageNet Results

(now human competitive on ImageNet)

Captioning Images

Labelling Results

Some good, some not-so-good

Reinforcement Learning

Google's DeepMind purchase

Learn to play games from the pixels alone

DeepMind Atari

Better than humans 2 hours after switching on

Reinforcement Learning

Google DeepMind's AlphaGo

Learn to play Go from (mostly) self-play

DeepMind AlphaGo Match 5

Basic Approach

  • Same as original Neural Networks in 1980s/1990s
  • Simple mathematical units ...
  • ... combine to compute a complex function

Single "Neuron"

One Neuron

Change weights to change output function

Multi-Layer

Layers of neurons combine and
can form more complex functions

Multi-Layer

Supervised Learning

  • while not done :
    • Pick a training case (xtarget_y)
    • Evaluate output_y from the x
    • Modify the weights so that output_y is closer to target_y for that x

Gradient Descent

Follow the gradient of the error
w.r.t the connection weights

Gradient-Descent

Workshop : SGD

    • Go to : http://ConvNetJS.com/
    • Look for : "Image 'painting'"
ConvNetJS Image Painting

Simple Network

ConvNetJS Painting : 5

Wider Network

ConvNetJS Painting : 20

Two-Ply Network

ConvNetJS Painting : 10+10

Deep Network and Time

ConvNetJS Painting : 10x7

"Hello World" → MNIST

  • Nice dataset from the late 1980s
  • Training set of 50,000 28x28 images
  • Now end-of-life as a useful benchmark

MNIST digits

Simple Network

Multi-Layer

... around 2-3% error rate on the test set

Workshop : MNIST

  • Go to : http://ConvNetJS.com/
  • Look for : "Classify MNIST"
  • ... CNN approach (rather than MLP)
ConvNetJS MNIST

Workshop : VirtualBox

  • Import Appliance 'fossasia ... .OVA'
  • Start the Virtual Machine...

VM : Console


Login: user
Password: password

#...

./run-jupyter.bash

#....
   

Workshop : Jupyter

  • On your 'host' machine
  • Go to http://localhost:8080/

VM : SSH

From your 'host' machine :


ssh -p 8282 user@localhost     
   

Workshop : TensorBoard

  • On your 'host' machine
  • Go to http://localhost:8081/

Theano

  • Optimised Numerical Computation in Python
  • Computation is described in Python code :
    • Theano operates on expression tree itself
    • Optimizes the tree for operations it knows
    • Makes use of numpy and BLAS
    • Also writes C/C++ or CUDA (or OpenCL)

0-TheanoBasics

1-MNIST-MLP

2-MNIST-CNN

Theano : detail

Theano diagram x1

... zoom out

Theano diagram x2

... zoom out

Theano diagram x3

Theano : "Simple RNN"

Theano diagram x4

New Problems

  • ImageNet Competition
  • over 15 million labeled high-resolution images...
  • ... in over 22,000 categories

ImageNet Karpathy

More Complex Networks

Google ImageNet

GoogLeNet (2014)

3-ImageNet-googlenet

  • Play with a pre-trained network

... Even More Complex

Google Inception v3

Google Inception-v3 (2015)

4-ImageNet-inception-v3

  • Play with a pre-trained network
  • NB: This may be painfully slow...

Need for Speed

CPU vs GPU

... need for GPU programmers

5-Commerce

6-Visual-Art

LSTM Units

LSTM

8-Natural-Language

Poetry : Epoch 1


JDa&g#sdWI&MKW^gE)I}<UNK>f;6g)^5*|dXdBw6m\2&XcXVy\ph8G<gAM&>e4+mv5}OX8G*Yw9&n3XW{h@&T\Fk%BPMMI
OV&*C_] ._f$v4I~$@Z^&[2
mOVe`4W)"L-KClkO]wu]\$LCNadyo$h;>$jV7X$XK#4,T(y"sa6W0LWf\'_{\#XD]p%ck[;O`!Px\#E>/Or(.YZ|a]2}q|@a9.g3nV,U^qM	$+:nlk0sd;V-Z&;7Y@Z "l-7P^C
"xBF~~{n} n\ Pcbc9f?=y)FIc1h5kvjIi
C<UNK>s	DWJr_$ZQtu"BTYm'|SMj-]Z<Vqj*.lh%IYW|q.GK:eNI"r>833?+RuUsOj_)a{\T}gH.zZR^(daC3mg5P0iFi]bqGo4?T|\>0_H&g889voTh=~)^DDRYND46z1J]x;<U>>%eNIRckL)N8n<UNK>n3i)+Ln8
?)9.#s7X]}$*sxZ"3tf ")
@'HW.;I5)C.*%}<jcNLN+Z__RWoryOb#
/`r
   

Poetry : Epoch 100


Som the riscele his nreing the timest stordor hep pIs dach suedests her, so for farmauteds?
By arnouy ig wore
Thou hoasul dove he five grom ays he bare as bleen,
The seend,
And, an neeer,
Whis with the rauk with, for be collenss ore his son froven faredure:
Then andy bround'd the CowE nom shmlls everom thoy men ellone per in the lave ofpen the way ghiind, thour eyes in is ple gull heart sind, I I wild,
Frreasuce anspeve, wrom fant beiver, not the afan
And in thou' histwish a it wheme-tis lating ble the liveculd;
Noorroint he fhallought, othelts.
   

Poetry : Epoch 1000


AWhis grook my glass' to his sweet,
Bub my fears liken?
And of live every in seedher;
A Lood stall,
But tare tought than thencer sud earth,
Use'st bee sechion,
For all exprit' are a daud in heaven doth her infook perust the fork the tent.

For maud,
The pittent gover
This and rimp,
Who new
  
Thoir oldes and did hards, cound.
   

Plays : Epoch 338

Larger network...


DEDENIUS	Why shoulmeying to to wife,
	And thou say: and wall you teading for
	that struke you down as sweet one.
	With be more bornow, bly unjout on the account:
	I duked you did four conlian unfortuned drausing-
	to sicgia stranss, or not sleepplins his arms
	Gentlemen? as write lord; gave sold.

AENEMUUNS	Met that will knop unhian, where ever have
	of the keep his jangst?icks he I love hide,
	Jach heard which offen, sir!'

	[Exit PATIIUS, MARGARUS arr	[Enter CLOTHUR]
   

Image Labelling

Image Labelling

Image Labels

Labelling Results

"A.I. Effect"

Wrap-up

  • Deep Learning may deserve some hype...
  • Getting the tools in one place is helpful
  • Having a GPU is VERY helpful
GitHub - mdda

* Please add a star... *

- QUESTIONS -


Martin.Andrews @
RedCatLabs.com


My blog : http://mdda.net/

GitHub : mdda