About Me

• Machine Intelligence / Startups / Finance
• • Moved from NYC to Singapore in Sep-2013
• 2014 = 'fun' :
• • Machine Learning, Deep Learning, NLP
  • Robots, drones
• Since 2015 = 'serious' :: NLP + deep learning
• • & GDE ML; TF&DL co-organiser
  • & Papers...
  • & Dev Course...

About Red Dragon AI

• Google Partner : Deep Learning Consulting & Prototyping
• SGInnovate/Govt : Education / Training
• Products :
• • Conversational Computing
  • Natural Voice Generation - multiple languages
  • Knowledgebase interaction & reasoning

Outline

• whoami = DONE
• What is VQA ?
• VQA models
• "Did the Model Understand the Question?"
• Demo of VQA ...
• ... showing some limitations
• Wrap-up

VQA

• Visual Question Answering
• MS COCO images (100k+)
• + crowd-sourced questions (700k+)
• Agrawal et al (2015)
• www.visualqa.org

VQA Samples

VQA Questions

VQA Network

• Example network described in original paper
• ~57% accuracy

VQA Modern Network

• Kazemi & Elqursh (2017) : Show, Ask, Attend, and Answer
• ~64% accuracy

Paper Time

• Did the Model Understand the Question?
  Mudrakarta et al (2018) :
• • Code available on GitHub - based on pytorch-vqa
  • which has a pre-trained VQA model (80Mb)
  • ... a bit finicky to set up (uses non-standard ResNet)
• Slightly earlier work : "Pathologies of Neural Models Make Interpretation Difficult"
  (previously "Right Answer for the Wrong Reason")
  Shi Feng et al (2018)

Paper Idea

• VQA performance looks great!
• • But is the model doing what we think it is?
• Test by changing the question :
• • Remove words
  • Add functionless phrases
• Similiar to adversarial attacks

Surprises

• If only question word allowed is "color" ...
• • model still more than 50% of original accuracy
• If all questions are *empty* ...
• • model still more than 44% of original accuracy
• More generally : Model doesn't care much about nouns

Distractors

• These phrases can kill performance :
• • In not a lot of words, ...
  • In not many words, ...
  • What is the answer to : ...
• But these phrases do no harm :
• • Answer this : ...
  • Answer this for me : ...
  • Tell me ...

Demo

• Whole notebook should auto-gather code + data

Deep Learning
MeetUp Group

• MeetUp.com / TensorFlow-and-Deep-Learning-Singapore
• Next Meeting :
• • 27-Sept, hosted at Google (guest : TPU.SG)
• Typical Contents :
• • Talk for people starting out
  • Something from the bleeding-edge
  • Lightning Talks

Deep Learning
Developer Course

• JumpStart module is Module #1 of 5
• Each 'module' will include :
• • Instruction
  • Individual Projects
  • 70%-100% funding via IMDA for SG/PR
• 1-2 Nov : Advanced Computer Vision
• 22-23 Nov : Advanced NLP
• Location : SGInnovate

Deep Learning : Jump-Start Workshop

• First part of Full Developer Course - last one oversubscribed
• Dates + Cost : Sept 12-13, S$600 (much less for SC/PR)
• • 2 week-days
  • Play with real models & Get inspired!
  • Pick-a-Project to do
  • Regroup on subsequent week-night(s)
• SGInnovate - Talent - Talent Development -
  Deep Learning Developer Series

RedDragon AI
Intern Hunt

• Opportunity to do Deep Learning all day
• Work on something cutting-edge
• Location : Singapore
• Status : SG/PR FTW
• Need to coordinate timing...