What is AdaGrad Optimizer in Deep Learning?

Introduction

AdaGrad (Adaptive Gradient Algorithm) is an optimization algorithm that adapts the learning rate for each parameter based on the frequency of its updates. It was introduced in the paper “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization” by Duchi et al. in 2011.

AdaGrad is particularly useful for dealing with sparse data or high-dimensional features, such as in natural language processing or recommender systems, where some parameters require more frequent updates than others.

How AdaGrad Works

AdaGrad adjusts the learning rate individually for each parameter by accumulating the square of past gradients: Let:

• $\theta_t$: Model parameters at time step $t$
• $g_t$: Gradient of the loss w.r.t. parameters at time $t$, i.e., $g_t = \nabla_\theta L(\theta_t)$
• $G_t$: Accumulated (element-wise) sum of squared gradients
• $\eta$: Initial learning rate (global)
• $\epsilon$: A small constant for numerical stability (typically $10^{-7}$) 

Then the AdaGrad algorithm proceeds with the following update steps:

1. Compute the average gradient over a mini-batch of size m:

2. Accumulate the squared gradients element-wise:

Here, G_t is a running total of the squared gradients, used to scale future updates. 

3. Compute the parameter update with a per-parameter learning rate:

This reduces the learning rate for parameters with large historical gradients, helping stabilize updates.

4. Apply the parameter update:

This final step updates the parameters using the scaled gradient, completing one iteration of AdaGrad.

Intuition

• Parameters that consistently receive large gradients will have their learning rates shrunk more aggressively.

• Parameters with sparse gradients maintain larger learning rates, making AdaGrad particularly useful for sparse feature spaces (e.g., in NLP or recommendation systems).

Code Example

This code defines and trains a simple Multi-Layer Perceptron (MLP) model using the AdaGrad optimizer for a single training step on synthetic data. The network consists of:

• An input layer of size 784 (e.g., 28×28 pixel image),
• One hidden layer with 128 neurons and a ReLU activation,
• An output layer with 10 units for multi-class classification (e.g., digits 0–9).

import torch
import torch.nn as nn
import torch.optim as optim

# Simple neural network
model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

# Loss function
criterion = nn.CrossEntropyLoss()

# Optimizer using AdaGrad
optimizer = optim.Adagrad(model.parameters(), lr=0.01, eps=1e-10)

# Dummy input and target
x = torch.randn(64, 784)  # batch of 64 samples
y = torch.randint(0, 10, (64,))  # random targets

# Training step
optimizer.zero_grad()
output = model(x)
loss = criterion(output, y)
loss.backward()
optimizer.step()

import torch
import torch.nn as nn
import torch.optim as optim

# Simple neural network
model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

# Loss function
criterion = nn.CrossEntropyLoss()

# Optimizer using AdaGrad
optimizer = optim.Adagrad(model.parameters(), lr=0.01, eps=1e-10)

# Dummy input and target
x = torch.randn(64, 784)  # batch of 64 samples
y = torch.randint(0, 10, (64,))  # random targets

# Training step
optimizer.zero_grad()
output = model(x)
loss = criterion(output, y)
loss.backward()
optimizer.step()

Advantages of AdaGrad

• Great for sparse data and high-dimensional inputs like NLP and recommendation systems.
• Helps make rapid progress on infrequent but informative features.

Disadvantages of AdaGrad

• The accumulated squared gradients keep growing over time, leading to a continuously shrinking learning rate.
• As a result, the learning rate may decay too aggressively, causing the training to stall prematurely (i.e., stop learning before convergence).
• Not ideal for long-term training or non-convex problems where continued updates are necessary.

When to Use AdaGrad

Use AdaGrad when:

• Working with sparse features (e.g., text data, one-hot encoded vectors). As AdaGrad naturally gives larger updates to infrequent features, which is ideal for sparse data.
• You expect some parameters to be updated far more frequently than others. Because AdaGrad scales learning rates inversely to the frequency of updates, benefiting rarely updated parameters.

Avoid AdaGrad when:

• You’re training deep models or non-convex problems for many epochs. Its learning rate shrinks aggressively, which can cause training to stall in complex architectures.
• You need a method that maintains learning capacity throughout training: AdaGrad’s accumulated gradient history can overly reduce learning rates; alternatives like Adam or RMSProp are better for long training cycles.

Video Explanation

• In this video, Neil Rhodes offers a detailed explanation of the AdaGrad algorithm, covering its motivation, mathematical formulation, and practical applications (Runtime: ~10 mins)

Related Articles:

• What are some options for making Backpropagation more efficient?
• Comparison of Optimization Algorithms in Deep Learning: SGD, Adam and more