Motivation

• CNNs : Well adapted to vision
• RNNs : Well adapted to sequences
• Need something for Relationships

... a reader piecing together evidence to predict the culprit in a murder-mystery novel ...

Sort-of-CLEVR

Non-Relational

• What is the shape of the red object? => Circle
• Is green object placed on the left side of the image? => Yes
• Is orange object placed on the upside of the image? => No

Sort-of-CLEVR

Relational

• What is the shape of the object closest to the red object? => Square
• What is the shape of the object furthest to the orange object? => Circle
• How many objects have same shape with the blue object? => 3

The (DeepMind) Paper

• "A simple neural network module
  for relational reasoning"
• Santoro, Raposo &
  Barrett, Malinowski, Pascanu, Battaglia, Lillicrap

Their Idea

• Allow network to create 'entity' nodes
• Combine pairs of nodes together
• • ... with a relationship detector
• Call this combo a Relation-Network (RN)

Misleading Diagram

• Seems to show smartness that isn't there

The Reality

• All-the-Things → $n$ nodes
• Examine all pairs of nodes : $O(n^2)$
• • Run $g_\theta()$ MLP over every combination
• Sum up resulting vectors
• A final $f_\phi()$ MLP to give 'answer'

This Talk : The Idea

Unpublished

• Model should isolate entities itself
• ... and then reason about them

Related Work

• Deep Mind : Relation Networks
• Hinton's Capsules
• Binding Problem

Model Outline

• Allow network to create 'entity' nodes
• Stream these as an 'internal dialog'
• • ( based on question )
• Reason about that stream
• • ... to answer the question
• Hence : Relationships from Entity Stream

Network Diagram

• Using same Sort-of-CLEVR experimental set-up

Some Code

• RFES : Implementation in PyTorch
• (RN and CNN-MLP from kimhc6028)

Results

Sort-of-CLVR

• Accuracy is higher than previous work
• Network is a fraction of size
• Training speed is comparable

Discussion

• What to do next?
• Constraint : ICLR deadline

" 1 more thing "

• Force single entity selection
• Delve deeper into representations
• Apply to MNIST 'patches'

TensorFlow
MeetUp Group

• MeetUp.com / TensorFlow-and-Deep-Learning-Singapore
• Next Meeting :
• • early-Nov-2017 : Hosted by Google
• Typical Contents :
• • Talk for people starting out
  • Something from the bleeding-edge
  • Lightning Talks

8-week Deep Learning
Developer Course

• Actual : Started in September
• Twice-weekly 3-hour sessions include :
• • Instruction
  • 3 structured projects
  • 2 self-directed projects
• Cost: S$3,000 (grants available)
• People are working hard...