Deep Q-Networks (DQN), Explained Like You are Building It

Introduction

In the Deep Q Learning notebook, you trained a simple Q-learning agent in a toy world. DQN (Deep Q-Network) is the next step: it keeps the same basic idea – learn how good each action is – but swaps the “Q-table” for a neural network, so it can handle much larger, messier inputs (like images). DQN is best known for learning Atari games from pixels and achieving strong performance across many games using one general approach. (Nature article: Human-level control through deep reinforcement learning by Mnih et al.)

This article walks through DQN in an intuitive way: what it’s trying to do, why it needs special tricks, what you would implement, and what DQN is good (and bad) at.

1) From Q-learning to DQN: what problem are we solving?

Q-learning in one sentence

Q-learning learns a number (Q(s,a)): “how good is it to take action (a) in state (s)?” Then it chooses the action with the highest number. (Springer article: Q-Learning by Watkins & Dayan)

In small problems, you can store those values in a table. But in real problems:

• A “state” might be a camera frame (thousands of pixels)
• There are too many states to have a table
• You need a function that generalizes: if two states look similar, the agent should behave similarly

DQN’s key idea

Replace the table with a neural network:

The network takes a state (s) and outputs a score for each action (a). That’s it – conceptually simple.

2) The DQN training target: “make today’s prediction match a better guess”

Title: DQN Training Flow
Source: AIML.com Research

DQN is built on temporal-difference (TD) learning: update your current guess using a slightly better guess. (Springer article: Learning to predict by the methods of temporal differences by Richard Sutton )

When the agent experiences a transition $((s,a,r,s’))$, we create a target value:

Then we train the network so that its current prediction ($Q_\theta(s,a)$) moves closer to ($y$). A common loss is:

Plain language:

• The network says: “this action is worth 0.4.”
• The environment says: “you got a reward now, and next you can probably get up to 0.7 more.”
• Update the network so its score becomes more realistic.

This is the same logic as tabular Q-learning but with neural nets.

3) Why DQN needs “stability tricks”

If you naively combine Q-learning + neural networks, training often becomes unstable. DQN’s success came from two major stabilizers described in the original work. (Nature article: Human-level control through deep reinforcement learning by Mnih et al)

Trick A: Experience Replay (learn from a shuffled “memory”)

The problem: consecutive experiences are highly correlated.
If you learn only from the most recent steps, your training data is like watching a movie one frame at a time – every frame is almost the same. Neural nets don’t love that.

Experience replay fixes this by storing transitions in a replay buffer:

Then, instead of training on the newest transition, you sample a random minibatch from memory. This makes the learning updates less “jittery” and more like standard supervised learning.

Replay-style ideas appeared in early RL systems; DQN made them standard practice for deep value-based RL. (Springer article: Self-improving reactive agents based on reinforcement learning, planning and teaching by Long-Ji Lin)

Upgraded replay:

Not all experiences are equally useful. Prioritized Experience Replay samples “surprising” or “important” transitions more often (roughly: big TD error => learn more from it). (arXiv paper: Prioritized Experience Replay by Schaul et al)

Trick B: Target Network (a “slow-moving teacher”)

The problem: your target ($y$) uses your own network’s predictions, so the target changes every time you learn. That can generate a feedback loop.

Target networks solve this by keeping a second network $(Q_{\theta^-})$ (the “teacher”) that updates slowly. In many implementations:

• Every $(N)$ steps: copy $(\theta \rightarrow \theta^-)$
• Or slowly blend parameters (“soft updates”)

This makes $(y)$ more stable, which makes training more stable. (Nature article: Human-level control through deep reinforcement learning by Mnih et al)

4) What you actually implement (the mental checklist)

A minimal DQN implementation usually has:

• Q-network $(Q_\theta)$ (neural net)
• Target network $(Q_{\theta^-})$
• Replay buffer (a list / ring buffer of transitions)
• Exploration policy (usually ε-greedy)
• Training loop (collect experience + gradient updates)

ε-greedy exploration (simple and effective)

Even if the network thinks action “Right” is best, you still sometimes try other actions:

• With probability ε: random action (explore)
• Otherwise: best action by $(Q_\theta(s,a))$ (exploit)

Early in training ε is high; later it decays. This exploration ensures that good but obscure strategies aren’t overlooked.

5) Dueling Networks: a smart architecture tweak (still DQN)

Sometimes, many actions are basically the same (e.g., in a quiet state, left vs right may not matter much). A dueling network makes learning more efficient by splitting the job into two parts:

• $(V(s))$: “how good is this state overall?”
• $(A(s,a))$: “how much better is action (a) than average in this state?”

Then it recombines them:

Intuition: the network can learn the value of being in a good/bad situation even before it learns the fine details of which action is best. This dueling architecture was introduced and tested for deep RL performance improvements in the Atari setting. (Proceedings of Machine Learning Research)

6) Where DQN works well (and where it doesn’t)

Good fits

• Discrete actions: move left/right, choose one of K options
• High-dimensional inputs: images, large sensor vectors, game states
• Problems where you can afford lots of interaction data (simulators are great)

The classic showcase was Atari via the Arcade Learning Environment benchmark. (jair.org)

Common limitations

• Not naturally for continuous actions (robot torque, steering angle). You typically move to policy-gradient / actor-critic methods for that
• Data hungry: needs many episodes/steps
• Sensitive to reward design: if rewards are sparse or poorly aligned, learning can stall or optimize the wrong thing (see the article on reward design)

7) Pros and Cons of DQNs

A famous fix for overestimation is Double DQN, which adapts Double Q-learning ideas to deep networks and reduces overestimation in DQN-style methods. (AAAI Open Access Journal)

8) “Modern DQN” in one paragraph: Rainbow

After DQN, researchers proposed many improvements (double Q-learning, prioritized replay, dueling networks, distributional value learning, better exploration, etc.). Rainbow is a well-known paper that studies combining several of these into one stronger agent. (AAAI Open Access Journal)

You don’t need Rainbow to understand DQN, but it’s useful to know that “DQN” in the wild often means “DQN plus a few upgrades.”

Takeaway

DQN is Q-learning that scales:

• Q-learning: action scores in a table
• DQN: action scores from a neural network
• Stability tools: experience replay + target networks
• Useful extension: dueling networks separate “state value” from “action advantage”

If you can explain those ideas clearly, you understand the core of one of the most influential deep RL algorithms.

Videos

• This tutorial “Simply Explaining Deep Q-Learning by Johnny Code”, contains step by step explanation, code walkthrough, and demo of how Deep Q-Learning (DQL) works, through Pytorch. The video also covers differences between Deep Q-Learning and Q-Learning. (Runtime: 34 mins 4 secs)

• This video, “Introduction to deep reinforcement learning by Serrano Academy”, explains Q-networks in reinforcement learning with policy gradients through examples and figures. (Runtime: 36 mins 25 secs)

References

• Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. (Nature)
• Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2013). Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602. (arXiv)
• Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013). The Arcade Learning Environment: An Evaluation Platform for General Agents. JAIR. (jair.org)
• Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8, 293–321. (Springer)
• Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. ICLR / arXiv:1511.05952. (arXiv)
• Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2016). Dueling Network Architectures for Deep Reinforcement Learning. ICML (PMLR). (Proceedings of Machine Learning Research)
• van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-learning. AAAI. (AAAI Open Access Journal)
• Hessel, M., Modayil, J., van Hasselt, H., et al. (2018). Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI. (AAAI Open Access Journal)
• Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning. (Springer)
• Sutton, R. S. (1988). Learning to predict by the methods of temporal differences.Machine Learning. (Springer)

Related articles on AIML.com:

1. What does ‘policy’ in Reinforcement Learning mean?
2. Introduction to Reinforcement Learning: A Beginner’s Guide
3. What are the Advantages and Disadvantages of Reinforcement Learning?

Author: Sam M, Brown University