The website is in Maintenance mode. We are in the process of adding more features.
Any new bookmarks, comments, or user profiles made during this time will not be saved.

Machine Learning Resources

What is an activation function? What are the different types of activation functions? Discuss their pros and cons

Bookmark this question

For a complete understanding of Neural Network, check out the post: Basic architecture of Neural Network, Training Process, and Hyper-parameters tuning

Mathematical model of a Neuron

Zooming into a neuron
Title: Zooming into a neuron – how is activation function applied
Source: V7 Labs

An activation function is a mathematical function applied to the output of a neuron (or node) in a neural network. When a neural network receives input data and processes it through its layers, each neuron computes a weighted sum of the inputs, adds a bias term, and then applies an activation function to produce the neuron’s output. This output is then passed to the next layer of neurons as input. Using a simple neural network example, the role of activation function is illustrated in the figure below:

A simple neural network

Role of Activation Function in a Neural Network
Title: Role of Activation Function in a Neural Network
Source: Research

Activation functions serve two main purposes:

  1. Introduce Non-linearity: Without non-linearity, a neural network would behave like a linear model, no matter how deep it is. Activation functions allow the network to learn and represent complex, non-linear mappings between inputs and outputs.
  2. Control Neuron Activation: Activation functions control the firing behavior of neurons. Depending on the activation function’s output, a neuron might become activated (output a non-zero value) or remain inactive (output zero).

Typically, same activation function is applied to all the hidden layers, while the output layer uses a different activation function, based on the type of prediction model aims to make. How to choose an activation function is explained later in this article.

Different types of Activation Functions:

Activation functions, which are popularly used in neural network models, are shown in the figure below.

Different types of Activation Functions:
Title: Different type of Activation Functions
Source: Stanford CS231 course

Other activations functions are Parametric ReLU (PReLU) (parametric version of Leaky ReLU), Scaled Exponential Linear Unit (SELU), Swish, Hard Swish, and Gaussian Error Linear Unit (GELU) Activation.

Pros and Cons of different Activation Functions

Title: Pros and Cons of Activation function
Source: Research

Which activation function to choose from?

Activation function for hidden layers:

In practice, the choice of activation functions is as follows in order of priority:

  • ReLU is the top choice as it is simpler, faster, much lower run time, better convergence performance and do not suffer from vanishing gradient issues
  • Leaky ReLU, PReLU, Maxout and ELU
  • tanh
  • Sigmoid (not preferred anymore)

Activation function for output layers:

  • Sigmoid for binary classification, multi-label classification
  • Softmax for multi-class classification
  • Identity / linear function for regression problems

Some key terms to know w.r.t Activation functions:


Related Question: What do you mean by saturation in neural network training? Discuss the problems associated with saturation

In the context of neural networks, saturation refers to a situation where the output of an activation function or neuron becomes very close to the function’s minimum or maximum value (asymptotic ends), and small changes in the input have little to no effect on the output. Saturation becomes a critical issue in neural network training as it leads to the vanishing gradient problem, limiting the model’s information capacity and its ability to learn complex patterns in the data. When a unit is saturated, small changes to its incoming weights will hardly impact the unit’s output. Consequently, a weight optimization training algorithm will face difficulty in determining whether this weight change positively or negatively affected the neural network’s performance. The training algorithm would ultimately reach a standstill, preventing any further learning from taking place.

Vanishing gradient

Related Question: What is the vanishing and exploding gradient problem, and how are they typically addressed?

Vanishing gradient refers to a problem that can occur during the training of deep neural networks, when the gradients of the loss function with respect to the model’s parameters become extremely small (close to zero) as they are backpropagated through the layers of the network during training. This leads to impairment in learning in deep neural networks (DNN). When the gradients become too small, it means that the model’s weights are not being updated effectively. As a result, the network’s training may stagnate or become extremely slow, making it difficult for the network to learn complex patterns in the data.

Zero-centered output

Related Question: Why is Zero-centered output preferred for an activation function?

Optimization algorithms like gradient descent tend to work more efficiently when gradients are centered around zero. Zero-centered activations ensure that the mean activation value is around zero, preventing them from becoming too small (vanishing gradients) or too large (exploding gradients). This contributes to smoother and faster convergence during training. In addition, zero-centered output helps with bias mitigation by ensuring that neurons start with balanced initial activations. Balanced activations can lead to more stable and unbiased updates to the model’s weights during training. However, please note that while zero centered property is important, it is not necessary.

Leave your Comments and Suggestions below:

Please Login or Sign Up to leave a comment

Partner Ad  

Find out all the ways
that you can

Explore Questions by Topics