Long Short-Term Memory with Quadratic Connections in Recursive Neural Networks for Representing Compositional Semantics
Long Short-Term Memory (LSTM) has been widely used in different applications, such as natural language processing, speech recognition, computer vision over Recurrent Neural Network (RNN) or Recursive Neural Network (RvNN) - a tree-structured RNN. Also, the LSTM-RvNN has been used to represent compositional semantics through the connections of hidden vectors over child units. However, the linear connections in existing LSTM networks are incapable of capturing complex semantic representations of natural language texts. For example, complex structures in natural language texts usually denote intricate relationships between words, such as negated sentiment or sentiment strengths. In the paper, quadratic connections of the LSTM model is proposed in terms of Recursive Neural Networks (abbreviated as qLSTM-RvNN) in order to attack the problem of representing compositional semantics. The proposed qLSTM-RvNN model is evaluated in the benchmark data sets containing semantic compositionality, i.e., sentiment analysis on Stanford Sentiment Treebank and semantic relatedness on SICK (Sentences Involving Compositional Knowledge) data set. Empirical results show that it outperforms state-of-the-art RNN, RvNN and LSTM networks in two semantic compositionality tasks by increasing classification accuracies and sentence correlation while significantly decreasing computational complexities.
Recurrent Neural Networks, Long Short-Term Memory Networks, Nonlinear Connections, Deep Learning.