Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Relationship detection based on object semantic inference and attention mechanisms

Zhang, L., Zhang, S., Shen, P., Zhu, G., Shah, S.A.A. and Bennamoun, M. (2019) Relationship detection based on object semantic inference and attention mechanisms. In: 2019 on International Conference on Multimedia Retrieval (ICMR) 2019, 10 - 13 June 2019, Ottawa ON, Canada

Link to Published Version:
*Subscription may be required


Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as <man ride horse>, <horse, carry, bag>. To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

Item Type: Conference Paper
Murdoch Affiliation(s): Information Technology, Mathematics and Statistics
Item Control Page Item Control Page


Downloads per month over past year