NeurIPS Lightning Talks
TensorFlow & Deep Learning SG
26 February 2019
About Me
 Machine Intelligence / Startups / Finance

 Moved from NYC to Singapore in Sep2013
 2014 = 'fun' :

 Machine Learning, Deep Learning, NLP
 Robots, drones
 Since 2015 = 'serious' :: NLP + deep learning

 & GDE ML; TF&DL coorganiser
 & Papers...
 & Dev Course...
About Red Dragon AI
 Google Partner : Deep Learning Consulting & Prototyping
 SGInnovate/Govt : Education / Training
 Products :

 Conversational Computing
 Natural Voice Generation  multiple languages
 Knowledgebase interaction & reasoning
Outline
whoami
= DONE
 The Talks :

 Neural ODEs
 Image correspondences
 Learning ImageNet layerbylayer
 Wrapup
Neural ODEs
 Mathematicians coming to DL
 Very different way of looking at NNs
 CoWinner of NeurIPS 2019 Best Paper
Foundation
 ResNets are common
 Each hidden layer is :

 a function of the previous one; PLUS
 a direct copy of the previous one
 For each layer :
output = layer(input) + input
 In mathematics : \( h_{t+1} = f(h_t, \theta_t) + h_t \)
The Idea
 \( h_{t+1} = f(h_t, \theta_t) + h_t \)
 \( h_{t+1}  h_t = f(h_t, \theta_t) \)
 \( h_{t+\delta}  h_t = f(h_t, \theta_t).\delta \) # Step a fraction of a layer
 \( {{dh_{t}}\over{dt}} = f(h_t, \theta_t, t) \)
 Suddenly, we have a Differential Equation!
Picture
So What?
 Differential Equations have :

 been studied for centuries
 well understood behaviours
 superefficient solvers
Still looks impractical...
 But we can train the parameters \( \theta_t \) ...

 to optimise our Loss function \( L() \)...
 by finding the gradients (as usual) ...
 ... using the adjoint sensitivity method (1962) !
 We already have nice
grad()
machinery, and modern ODE solvers
In a nutshell
 The resulting algorithm is memory and time efficient
 Can explicitly trade off accuracy for speed
Possibilities
 Moving to 'continuous layers' lets us :

 Do an RNN at irregular time intervals
 Cope with missing data easily
 Create Normalising flows (~ inverting a NN)
Invertible Flows
Summary
 Illustrates how Mathematicians "Think Different"
 ... and opens up new possibilities
 Code on GitHub
Image correspondences
 One 'standardly impressive' paper
 One 'crazy impressive' paper
Model in a Picture
 Losses for finding points (based on groundtruth), and being geometrically consistent
Model in a Picture
 Amazing thing : Weakly supervised training
Weak Supervision
 Undersold (IMHO) in the paper itself
 The training was only supervised via :

 This is a cat : This is another cat
 This is a cat : This is not a cat
 ⇒ Learn to map the cat keypoints
 With this 'weak supervision', model still learns
Summary
 Excellent techniques shown at NeurIPS ...
 ... being surpassed by crazier techniques
 Which also open up new possibilities
Learning ImageNet
layerbylayer
 This shouldn't be possible
 Contradicts lots of accepted wisdom
 Lots of avenues for research
Model in a Picture
 Freeze weights when moving on to next layer
Training Accuracy
 Even 1layer ImageNet is beneficial ...
Lots of Ideas
 Fullmodel training not essential
 This procedure :

 Does not use (much) more computation (can cache results)
 Proves that a bad brain can be improved layerwise
 Could allow 'compression' as the model is built
 Still early days for the implications, though

Summary
 Still areas ripe for research
 Question everything ...
 ... including academic ratrace
Wrapup
 NeurIPS was in Montréal, in December
 Already there is new stuff coming along
 Looking forwards to more in 2019!
* Please add a star... *
Deep Learning
MeetUp Group
Deep Learning : JumpStart Workshop
Deep Learning
Developer Course
 Module #1 : JumpStart (see previous slide)
 Each 'module' will include :

 Indepth instruction, by practitioners
 Individual Projects
 70%100% funding via IMDA for SG/PR
 Stay informed :
http://bit.ly/rdaicourses2019
 Location : SGInnovate/BASH
RedDragon AI
Intern Hunt
 Opportunity to do Deep Learning all day
 Work on something cuttingedge
 Location : Singapore
 Status : SG/PR FTW
 Need to coordinate timing...
Conversational AI & NLP
MeetUp
http://bit.ly/convaisg
 Next Meeting : Date TBA, hosted at TBD
 Typical Contents :

 Applicationcentric talks
 Talks with technical content
 Lightning Talks
 Target : >2 Members !!