Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Bi-SAN-CAP: Bi-Directional Self-Attention for Image Captioning

Hossain, M.Z., Sohel, F., Shiratuddin, M.F., Laga, H.ORCID: 0000-0002-4758-7510 and Bennamoun, M. (2019) Bi-SAN-CAP: Bi-Directional Self-Attention for Image Captioning. In: Digital Image Computing: Techniques and Applications (DICTA) 2019, 2 - 4 December 2019, Hyatt Regency Perth, Australia

Link to Published Version:
*Subscription may be required


In a typical image captioning pipeline, a Convolutional Neural Network (CNN) is used as the image encoder and Long Short-Term Memory (LSTM) as the language decoder. LSTM with attention mechanism has shown remarkable performance on sequential data including image captioning. LSTM can retain long-range dependency of sequential data. However, it is hard to parallelize the computations of LSTM because of its inherent sequential characteristics. In order to address this issue, recent works have shown benefits in using self-attention, which is highly parallelizable without requiring any temporal dependencies. However, existing techniques apply attention only in one direction to compute the context of the words. We propose an attention mechanism called Bi-directional Self-Attention (Bi-SAN) for image captioning. It computes attention both in forward and backward directions. It achieves high performance comparable to state-of-the-art methods.

Item Type: Conference Paper
Murdoch Affiliation: College of Science, Health, Engineering and Education
Item Control Page Item Control Page