# RelationshipsfromEntity Stream

#### PyTorch & Deep Learning SG

19 October 2017

• Machine Intelligence / Startups / Finance
• Moved from NYC to Singapore in Sep-2013
• 2014 = 'fun' :
• Machine Learning, Deep Learning, NLP
• Robots, drones
• Since 2015 = 'serious' :: NLP + deep learning
• & Papers...
• & Dev Course...

## Motivation

• CNNs : Well adapted to vision
• RNNs : Well adapted to sequences
• Need something for Relationships
... a reader piecing together evidence to predict the culprit in a murder-mystery novel ...

## Sort-of-CLEVR

#### Non-Relational

• What is the shape of the red object? => Circle
• Is green object placed on the left side of the image? => Yes
• Is orange object placed on the upside of the image? => No

## Sort-of-CLEVR

#### Relational

• What is the shape of the object closest to the red object? => Square
• What is the shape of the object furthest to the orange object? => Circle
• How many objects have same shape with the blue object? => 3

## The (DeepMind) Paper

• "A simple neural network module
for relational reasoning"
• Santoro, Raposo &
Barrett, Malinowski, Pascanu, Battaglia, Lillicrap

 https://arxiv.org/abs/1706.01427 

## Their Idea

• Allow network to create 'entity' nodes
• Combine pairs of nodes together
• ... with a relationship detector
• Call this combo a Relation-Network (RN)

• Seems to show smartness that isn't there

## The Reality

• All-the-Things → $n$ nodes
• Examine all pairs of nodes : $O(n^2)$
• Run $g_\theta()$ MLP over every combination
• Sum up resulting vectors
• A final $f_\phi()$ MLP to give 'answer'

## This Talk : The Idea

#### Unpublished

• Model should isolate entities itself
• ... and then reason about them

## Model Outline

• Allow network to create 'entity' nodes
• Stream these as an 'internal dialog'
• ( based on question )
• ... to answer the question
• Hence : Relationships from Entity Stream

## Network Diagram

• Using same Sort-of-CLEVR experimental set-up

## Some Code

 https://github.com/mdda/relationships-from-entity-stream 

## Results

#### Sort-of-CLVR

• Accuracy is higher than previous work
• Network is a fraction of size
• Training speed is comparable

## Discussion

• What to do next?

## " 1 more thing "

• Force single entity selection
• Delve deeper into representations
• Apply to MNIST 'patches'

## Wrap-up

• Deep Learning papers are very readable
• Cutting edge experiment runs in <1 hour
• PyTorch is great for exploring new ideas

## 8-week Deep LearningDeveloper Course

• Actual : Started in September
• Twice-weekly 3-hour sessions include :
• Instruction
• 3 structured projects
• 2 self-directed projects
• Cost: S\$3,000 (grants available)
• People are working hard...

# - QUESTIONS -

### Martin.Andrews @ RedDragon.ai

My blog : http://mdda.net/

GitHub : mdda