Neural Motifs: Scene Graph Parsing with Global Context (CVPR 2018)

by Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi

Overview

• In this work, we investigate the problem of producing structured graph representations of visual scenes. Similar to object detection, we must predict a box around each object. Here, we also need to predict an edge (with one of several labels, possibly background) between every ordered pair of boxes, producing a directed graph where the edges hopefully represent the semantics and interactions present in the scene.
• We present an analysis of the Visual Genome Scene Graphs dataset. In particular:
  • Object labels (e.g. person, shirt) are highly predictive of edge labels (e.g. wearing), but not vice versa.
  • Over 90% of the edges in the dataset are non-semantic.
  • There is a significant amount of structure in the dataset, in the form of graph motifs (regularly appearing substructures).
• Motivated by our analysis, we present a simple baseline that outperforms previous approaches.
• We introduce Stacked Motif Networks (MotifNet), which is a novel architecture that is designed to capture higher order motifs in scene graphs. In doing so, it achieves a sizeable performance gain over prior state-of-the-art.

Read the paper!

The current version is available on arxiv.

Bibtex

@inproceedings{zellers2018scenegraphs,
  title={Neural Motifs: Scene Graph Parsing with Global Context},
  author={Zellers, Rowan and Yatskar, Mark and Thomson, Sam and Choi, Yejin},
  booktitle = "Conference on Computer Vision and Pattern Recognition",  
  year={2018}
}

View some examples!

Check out this tool I made to visualize the scene graph predictions. Disclaimer: the predictions are from an earlier version of the model, but hopefully they’re still helpful!

Code

Visit the neural-motifs GitHub repository for our reference implementation and instructions for running our code.

It is released under the MIT license.

Checkpoints

• Pretrained Detector
• Motifnet-SGDet
• Motifnet-SGCls/PredCls
• Baseline-SGDet - note: this is the baseline that uses image features. For Freq/Freq-overlap, all you need is the Pretrained Detector checkpoint.
• Baseline-SGCls/PredCls

Comments/ questions?

Feel free to contact me or check out my websiteif you have any more questions!