Q-Learning: Teaching an Agent by Building a “Scoreboard” for Actions

Imagine you’re learning to navigate a new city. At every intersection you can choose left, right, or straight. Some choices lead you closer to your destination; others waste time or take you into dangerous areas. Over time, you build intuition like:

• “From this intersection, turning right usually works.”
• “From that intersection, going straight is a bad idea.”

Q-learning is basically that: it learns a scoreboard of how good each action is in each situation, then uses that scoreboard to choose actions better and better.

This post explains:

• what Q-learning learns,
• how its training update works,
• how exploration fits in,
• real-world-style use cases,
• pros/cons,
• and what Double Q-learning is (and why it exists).

1) The idea in one line

Q-learning learns a function $Q(s,a)$:

“If I am in state (s) and I take action (a), how good will that be in the long run?”

“Good” means expected total future reward, not just the next reward.

2) A tiny example: a delivery rider choosing routes

Let’s use a grounded toy example:

A delivery rider in a city chooses between two routes:

• Route A: usually fast, sometimes blocked by traffic
• Route B: slower but reliable

The rider’s “state” might include:

• time of day (rush hour or not),
• current area,
• weather.

The “action” is which route to take.

The “reward” could be:

• +1 for a successful on-time delivery,
• −0.01 per minute of delay,
• −1 if late.

At first, the rider doesn’t know which route is better in each situation. Q-learning helps the agent learn a scoreboard like:

• $Q(\text{rush hour}, \text{Route A}) = 0.3$
• $Q(\text{rush hour}, \text{Route B}) = 0.6$

Then it chooses the higher one more often.

3) The key components of Q-learning

States (s)

A “state” is whatever information the agent uses to decide what to do next.
In a gridworld: state might be “my current cell.”
In delivery: state might be “location + time + weather.”

Actions (a)

What choices are available.
In gridworld: up/down/left/right.
In delivery: route A/B.

Rewards (r)

Numbers that define what you want (profit, speed, safety, etc.). (And as you saw in the rewards article: reward design is hard.)

Q-table (Q(s,a))

In the simplest version, Q-learning stores a table of numbers:

• one row per state,
• one column per action.

Each entry is the agent’s current guess of how good that action is in that state.

4) How Q-learning updates its “scoreboard”

Every step, the agent observes a transition:

Meaning:

• I was in state $(s_t)$
• I took action $(a_t)$
• I got reward $(r_{t+1})$
• I ended up in state $(s_{t+1})$

Q-learning updates its score like this:

This looks scary until you read it as plain English:

1. Keep your old belief: $Q(s_t,a_t)$
2. Compute a better target:
   • reward you just got: $(r_{t+1})$
   • plus the best future score from the next state: $(\gamma \max_{a’}Q(s_{t+1},a’))$
3. Move a small step toward that target $(controlled by (\alpha))$

What do $\alpha$ and $\gamma$ mean?

• $\alpha$ is the learning rate: how quickly you update from new experience
  • too high: learning can bounce around
  • too low: learning is slow
• $\gamma$ is the discount: how much you care about the future
  • close to 1: long-term planning
  • smaller: more short-term focus

5) Exploration: how the agent discovers better actions

If the agent always picks the action with the highest Q-score, it might get stuck early: it could find a “pretty good” route and never discover a better one.

The simplest exploration strategy is ε-greedy:

• With probability $\varepsilon$: choose a random action (explore)
• Otherwise: choose $\arg\max_a Q(s,a)$ (exploit)

Early in training, $\varepsilon$ is often large (explore a lot), then decays over time.

6) The training loop (in words)

Q-learning training is repetitive and simple:

1. Start an episode (or start at a state)
2. Choose an action (with exploration)
3. Observe reward and next state
4. Update $Q(s,a)$
5. Repeat the above steps

Over many experiences, the Q-table becomes a useful “map” of what to do.

Under standard assumptions (like visiting state-action pairs enough and decaying learning rates), tabular Q-learning has convergence guarantees—one reason it’s such a classic algorithm.

7) Real-world-flavored applications

Q-learning is easiest when states and actions are discrete (or can be discretized). Historically, RL (including methods closely related to Q-learning) has been applied in areas like:

• Network routing (choosing where to send packets when network conditions change)
• Scheduling/control problems like improving elevator dispatching policies

Today, pure tabular Q-learning is mostly used for:

• education,
• prototyping,
• small decision systems,
• building intuition before deep RL.

8) Pros and Cons

Pros

• Simple and interpretable: the Q-table is a readable scoreboard
• Model-free: you don’t need to know the transition probabilities
• Strong foundation: many modern algorithms are “Q-learning + extras”

Cons

• Doesn’t scale to huge state spaces: tables blow up
• Hard with continuous actions: “max over actions” becomes tricky
• Can be sample-inefficient: may need many trials
• Function approximation can be unstable: when you replace the table with a flexible function (like a neural net), learning can become less stable and can suffer from value overestimation issues

Those limitations motivate algorithms like DQN.

9) What is Double Q-learning?

A subtle problem in Q-learning comes from this part:

Taking a max over noisy estimates tends to create overestimation bias: even if each action’s estimate is “wrong in random directions,” the largest one is often “wrong in the optimistic direction.”

Double Q-learning addresses this by maintaining two value estimators and separating:

• which action seems best (selection),
• how good that action really is (evaluation).

The classic Double Q-learning paper shows this reduces overestimation and can improve learning behavior.

Intuition:

Instead of trusting one scoreboard to both pick the best action and judge it, you use two scoreboards so errors don’t reinforce themselves as strongly.

This idea later inspires Double DQN in deep reinforcement learning.

Takeaway

Q-learning is one of the clearest RL algorithms:

• It learns a scoreboard $Q(s,a)$
• It improves by nudging scores toward “reward now + best future score”
• With exploration, it discovers better actions over time
• Double Q-learning fixes a common “too optimistic” bias caused by the max operator

If you understand Q-learning, you’ve built the foundation needed to understand DQN and many other RL methods.

Videos

• This video by Steve Brunton describes Q Learning, temporal difference learning, and other temporal difference learning algorithms like SARSA, as well as how it connects to dopamine in biology.
  (Runtime: 13 mins – watch from 22:50, or 22 mins – watch from 13:00 for broader algorithm understanding on TD learning)

• This video by CodeEmporium discusses Q Learning and topics like value based functions and Q-value. (Runtime: 12 mins)

References

• Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279–292.
• Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9–44.
• Tsitsiklis, J. N. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning, 16, 185–202.
• Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6), 1185–1201.
• Thrun, S., & Schwartz, A. (1993). Issues in using function approximation for reinforcement learning. (Overestimation/instability issues with approximation.)
• van Hasselt, H. (2010). Double Q-learning. NeurIPS 23, 2613–2621.
• Crites, R. H., & Barto, A. G. (1996). Improving elevator performance using reinforcement learning. NeurIPS.
• Boyan, J. A., & Littman, M. L. (1993).Packet routing in dynamically changing networks: A reinforcement learning approach.NeurIPS.

Related articles on AIML.com:

1. Introduction to Reinforcement Learning: A Beginner’s Guide
2. What does ‘policy’ in Reinforcement Learning mean?
3. What are the Advantages and Disadvantages of Reinforcement Learning?

Author: Sam M, Brown University