Catalog Home Page

Relationship detection based on object semantic inference and attention mechanisms

Zhang, L., Zhang, S., Shen, P., Zhu, G., Shah, S.A.A. and Bennamoun, M. (2019) Relationship detection based on object semantic inference and attention mechanisms. In: 2019 on International Conference on Multimedia Retrieval (ICMR) 2019, 10 - 13 June 2019, Ottawa ON, Canada

[img]
Preview
Link to Published Version: https://doi.org/10.1145/3323873.3325025
*Subscription may be required

Abstract

Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as <man ride horse>, <horse, carry, bag>. To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

Item Type: Conference Paper
Murdoch Affiliation: Information Technology, Mathematics and Statistics
URI: http://researchrepository.murdoch.edu.au/id/eprint/48527
Item Control Page Item Control Page

Downloads

Downloads per month over past year