An Attention Module for Convolutional Neural Networks

Baozhou Zhu*, Peter Hofstee, Jinho Lee, Zaid Al-Ars

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

8 Downloads (Pure)

Abstract

Attention mechanism has been regarded as an advanced technique to capture long-range feature interactions and to boost the representation capability for convolutional neural networks. However, we found two ignored problems in current attentional activations-based models: the approximation problem and the insufficient capacity problem of the attention maps. To solve the two problems together, we initially propose an attention module for convolutional neural networks by developing an AW-convolution, where the shape of attention maps matches that of the weights rather than the activations. Our proposed attention module is a complementary method to previous attention-based schemes, such as those that apply the attention mechanism to explore the relationship between channel-wise and spatial features. Experiments on several datasets for image classification and object detection tasks show the effectiveness of our proposed attention module. In particular, our proposed attention module achieves 1.00 % Top-1 accuracy improvement on ImageNet classification over a ResNet101 baseline and 0.63 COCO-style Average Precision improvement on the COCO object detection on top of a Faster R-CNN baseline with the backbone of ResNet101-FPN. When integrating with the previous attentional activations-based models, our proposed attention module can further increase their Top-1 accuracy on ImageNet classification by up to 0.57 % and COCO-style Average Precision on the COCO object detection by up to 0.45. Code and pre-trained models will be publicly available.

Original languageEnglish
Title of host publicationArtificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings
EditorsIgor Farkaš, Paolo Masulli, Sebastian Otte, Stefan Wermter
PublisherSpringer
Pages167-178
Number of pages12
Volume12891
ISBN (Print)9783030863616
DOIs
Publication statusPublished - 2021
Event30th International Conference on Artificial Neural Networks, ICANN 2021 - Virtual, Online at Bratislava, Slovakia
Duration: 14 Sep 202117 Sep 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12891 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference30th International Conference on Artificial Neural Networks, ICANN 2021
Country/TerritorySlovakia
CityVirtual, Online at Bratislava
Period14/09/2117/09/21

Keywords

  • Attention mechanism
  • Convolution
  • Representation

Fingerprint

Dive into the research topics of 'An Attention Module for Convolutional Neural Networks'. Together they form a unique fingerprint.

Cite this