Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Text to image synthesis for improved image captioning

Hossain, Md.Z., Sohel, F., Shiratuddin, M.F., Laga, H.ORCID: 0000-0002-4758-7510 and Bennamoun, M. (2021) Text to image synthesis for improved image captioning. IEEE Access, 9 . pp. 64918-64928.

PDF - Published Version
Download (1MB) | Preview
Free to read:
*No subscription required


Generating textual descriptions of images has been an important topic in computer vision and natural language processing. A number of techniques based on deep learning have been proposed on this topic. These techniques use human-annotated images for training and testing the models. These models require a large number of training data to perform at their full potential. Collecting human generated images with associative captions is expensive and time-consuming. In this paper, we propose an image captioning method that uses both real and synthetic data for training and testing the model. We use a Generative Adversarial Network (GAN) based text to image generator to generate synthetic images. We use an attention-based image captioning method trained on both real and synthetic images to generate the captions. We demonstrate the results of our models using both qualitative and quantitative analysis on popularly used evaluation metrics. We show that our experimental results achieve two fold benefits of our proposed work: i) it demonstrates the effectiveness of image captioning for synthetic images, and ii) it further improves the quality of the generated captions for real images, understandably because we use additional images for training.

Item Type: Journal Article
Murdoch Affiliation(s): Information Technology, Mathematics and Statistics
Publisher: IEEE
Copyright: © 2021 IEEE
Item Control Page Item Control Page


Downloads per month over past year