Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Language modeling through Long-Term memory network

Nugaliyadde, A., Sohel, F., Wong, K.W. and Xie, Hong (2019) Language modeling through Long-Term memory network. In: International Joint Conference on Neural Networks (IJCNN) 2019, 14 - 19 July 2019, Budapest, Hungary

Link to Published Version: https://doi.org/10.1109/IJCNN.2019.8851909
*Subscription may be required

Abstract

Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient problems. While LSTM and other memory networks address this problem, they are not capable of handling long sequences (50 or more data points long sequence patterns). Language modelling requiring learning from longer sequences are affected by the need for more information in memory. This paper introduces Long Term Memory network (LTM), which can tackle the exploding and vanishing gradient problems and handles long sequences without forgetting. LTM is designed to scale data in the memory and gives a higher weight to the input in the sequence. LTM avoid overfitting by scaling the cell state after achieving the optimal results. The LTM is tested on Penn treebank dataset, and Text8 dataset and LTM achieves test perplexities of 83 and 82 respectively. 650 LTM cells achieved a test perplexity of 67 for Penn treebank, and 600 cells achieved a test perplexity of 77 for Text8. LTM achieves state of the art results by only using ten hidden LTM cells for both datasets.

Item Type: Conference Paper
Murdoch Affiliation: College of Science, Health, Engineering and Education
URI: http://researchrepository.murdoch.edu.au/id/eprint/52104
Item Control Page Item Control Page