Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning

Sharif, N., Jalwana, M.A.A.K., Bennamoun, M., Liu, W. and Shah, S.A.A. (2020) Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning. In: 35th International Conference on Image and Vision Computing New Zealand (IVCNZ), 25 - 27 November 2020, Wellington, New Zealand

Link to Published Version: https://doi.org/10.1109/IVCNZ51579.2020.9290719
*Subscription may be required

Abstract

Image captioning is a challenging vision-to-language task, which has garnered a lot of attention over the past decade. The introduction of Encoder-Decoder based architectures expedited the research in this area and provided the backbone of the most recent systems. Moreover, leveraging relationships between objects for holistic scene understanding, which in turn improves captioning, has recently sparked interest among researchers. Our proposed model encodes the spatial and semantic proximity of object pairs into linguistically-aware relationship embeddings. Moreover, it captures the global semantics of the image using NASNet. This way, true semantic relations that are not apparent in visual content of an image can be learned, such that the decoder can attend to the most relevant object relations and visual features to generate more semantically-meaningful captions. Our experiments highlight the usefulness of linguistically-aware object relations as well as NASNet visual features for image captioning.

Item Type: Conference Paper
Murdoch Affiliation(s): IT, Media and Communications
URI: http://researchrepository.murdoch.edu.au/id/eprint/59413
Item Control Page Item Control Page