The website is in Maintenance mode. We are in the process of adding more features.
Any new bookmarks, comments, or user profiles made during this time will not be saved.

Machine Learning Resources

What is Multi-head Attention and how does it improve model performance over single Attention head?

Bookmark this question

Related Questions:
– Explain Self-Attention, and Masked Self-Attention as used in Transformers
– What are transformers? Discuss the major breakthroughs in transformer models
– Explain the Transformer Architecture

Multi-head attention extends the idea of single-head attention by running multiple attention heads in parallel on the same input sequence. This allows the model to learn different types of relationships and patterns within the input data simultaneously, thereby considerably enhancing the expressive power of the model as compared to using just single attention head.

Related Question: Explain Attention, and Masked Self-Attention as used in Transformers

Self-Attention vs Multi-head Attention
Title: (left) Single Attention head (also known as Scaled Dot-Product Attention)
(right) Multi-Head Attention consists of several attention layers running in parallel
Source: Attention is all you need (2017)

Implementation of Multi-Head Attention

  1. Instead of having a single set of learnable K, Q, and V matrices, multiple sets are initialized (one for each attention head)
  2. Each attention head independently computes attention scores and produces its own attention weighted output.
  3. The outputs from all the attention heads are concatenated and passed through a linear transformation to create the final multi-head attention output.
  4. The key innovation is that each attention head may focus on different parts of the input, thereby capturing various patterns and relationships within the data.

Benefits and Limitations of Multi-Head Attention over Single-Head Attention

   – Increased Expressiveness: Multi-head attention allows the model to capture different types of dependencies and patterns simultaneously. This is crucial for understanding complex relationships in the data.

   – Improved Generalization: By learning multiple sets of attention parameters, the model becomes more robust and adaptable to different tasks and datasets.

Increased Computational Complexity: While multi-head attention enhances the model’s capabilities, it also increases the computational complexity, thereby needing more compute resources. To help mitigate this, during inference time, a mechanism called Head Pruning is employed to discard heads that are less useful.

Leave your Comments and Suggestions below:

Please Login or Sign Up to leave a comment

Partner Ad  

Find out all the ways
that you can

Explore Questions by Topics

Partner Ad

Learn Data Science with Travis - your AI-powered tutor |