Research in CS Presentation

Title: Evaluation of Attention Mechanisms for Recurrent Neural Networks

Presenter: Alexander Katrompas

Advisor: Dr. Vangelis Metsis

Time/Date: Friday, May 7th @12:30 pm – 1:15 pm (CDT)


Zoom Link:



Attention mechanisms were first introduced into neural network architectures in 2015. Since then, many forms of attention have been introduced. All attention mechanisms have one common purpose; they attempt to create a focus within a network to enhance prediction by “attending” to parts of the input that otherwise would not be recognized as strongly. These enhancements may or may not have a different target optimization than the neural network itself, which varies with the type of attention mechanism and the type of training.

Attention has quickly gained ground in the areas of text analysis and image recognition. Originally, attention was applied to various RNN and CNN architectures and was shown to improve prediction. This technique has proven so successful that in the case of text processing, it has also been shown to stand alone without the need for RNN or CNN layers. The success of this stand-alone approach has been specifically and successfully applied to text and image prediction domains.

This study seeks to investigate and understand the various popular attention mechanisms with time-series data which, unlike text and image prediction, has an explicit time-dependent relationship between sequences. This study also seeks to build on a previous time-series study

investigating one specific attention mechanism, self-attention, so as to compare and contrast those results with other attention mechanisms and to further understand the relationship between time-series data and attention mechanisms.

Deadline: May 8, 2021, midnight