Introduction to Reinforcement Learning: A Beginner’s Guide

Introduction

Reinforcement Learning (RL) is a powerful branch of machine learning that enables an agent to learn how to make decisions through trial and error. Unlike other machine learning methods, such as supervised learning or unsupervised learning, RL focuses on learning from feedback (rewards or penalties) based on the agent’s actions. The key difference among three methods can be referred as following:

• Supervised learning requires labeled data, focusing on mapping inputs to outputs.

• Unsupervised learning seeks to uncover patterns or structures in unlabeled data, with no predefined outcome.

• Reinforcement learning involves decision-making over time, where an agent learns through trial and error by interacting with an environment to maximize long-term rewards.

A more comprehensive comparison among all three methods shown in table below.

The ability to learn from experience, without being told exactly what to do, makes RL an exciting and powerful approach in varying research directions. Think of RL as how humans and animals learn: trial and error. For instance, a child learning to ride a bike falls and gets back up, adjusting their actions based on the feedback they get (pain, joy, or success). Similarly, RL agents learn to make better decisions by receiving feedback on their actions, and refining their approach as they go.

In this article, we will explore the fundamentals of reinforcement learning, its real-world applications, and why it’s one of the most promising fields in AI today.

Core Components of Reinforcement Learning

To understand RL more deeply, let’s break down its key components:

• The Agent: This is the decision-maker. It observes the environment, decides on actions, and tries to maximize rewards. Imagine it as the player in a game or a child in the bike-learning process. 

• The Environment: This is everything the agent interacts with. It could be a virtual world (like a video game) or the physical world (like a robot navigating a room). The environment gives feedback to the agent based on the actions it takes.

• State space (S）: The state $s \in S$ represents the current situation of the environment. In a game of chess, the state could include the position of all the pieces on the board. It’s a snapshot of everything the agent can observe at any given time. Denote $s_t$ as the state at time step $t$.

• Action space (A）: An action $a \in A$ is the decision the agent makes in a given state. In chess, this would be a move, such as moving a pawn forward or castling the king. Denote $a_t$ as the action at time step $t$.

• Reward  (R）: After each action, the agent receives feedback from the environment in the form of a reward (positive feedback) or penalty (negative feedback). The reward helps the agent understand whether its action was good or bad. We denote $R_t$ as the reward at time step $t$.

Note that the definition of the agent in reinforcement learning is more ancient and is different from AI agents from foundation models/generative AI perspectives. The scope of RL agent is specifically for the algorithm solving sequential decision making with feedback. The details of AI agents, which are widely used, can be learned from the blog from IBM: What are AI agents?

Learning Process

The whole components can be combined together as a closed-loop control framework in Figure above. The agent serves as a controller from the control theory perspective with input and output. This process is modeled with the Markov decision process (MDP), where each interaction between the agent and the environment as one cycle refers to a single time step. The agent starts in an initial state and begins interacting with the environment. Then for each following time step, the agent will deploy an action to the environment given the current state, and the environment will transition to the next state and provide the corresponding reward feedback (if the sign is negative, one can view it as a penalty). Formally, the goal of the agent is to maximize the expected cumulative reward (also called return) over time.

The return $G_t$ at any time step $t$ is the sum of the rewards the agent receives, discounted by a factor $\gamma$ (where $0 \leq \gamma \leq 1$), which determines the present value of future rewards. By adjusting $\gamma$, we can control how much the agent should value future rewards compared with immediate ones.

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

, where $R_{t+k}$ is the reward received at time step $t+k$. The detailed reasons for incorporating a discount factor in the return will be discussed in the following article.

Real-World Applications of Reinforcement Learning

RL has shown immense promise in diverse fields. Some of its groundbreaking applications include:

• Gaming: RL algorithms like AlphaGo by DeepMind have revolutionized AI’s performance in complex games like Go, beating human champions by learning optimal strategies through self-play.

• Robotics and Autonomous Vehicles: RL powers robots that can learn to navigate and manipulate objects, as well as self-driving cars that make real-time decisions to navigate complex environments.

• Healthcare: In personalized medicine, RL is used to optimize drug dosages, personalize treatment plans, and even assist in robotic surgeries, improving patient outcomes.

• Business: RL enhances recommendation systems, improves supply chain logistics, and refines financial predictions by learning from vast amounts of data and continuously improving decisions.

• AI Model Training: RL is also used in fine-tuning AI models like ChatGPT, where human feedback guides the learning process, ensuring the model generates more accurate and contextually relevant responses. This technique, known as Reinforcement Learning from Human Feedback (RLHF), is crucial for improving the alignment of AI systems with human expectations.

Evolution of Reinforcement Learning

Concurrent Developments (1900s)

• Trial-and-Error Learning (1900s):
  • Thorndike’s Law of Effect (1911) defines trial-and-error learning as the basis for RL.
  • Minsky (1954) introduced SNARCs, an early model for RL systems.

• Optimal Control (1950s–1960s):
  • Bellman (1957) developed the Bellman equation, foundational for RL’s dynamic programming.
  • Markov Decision Processes (MDPs) introduced in 1957, further developed by Howard (1960).

• Temporal Difference Learning (1950s–1970s):
  • Rooted in psychology (Thorndike, 1911), this approach started with Minsky (1954) and was expanded by Samuel (1959).
  • Klopf’s models (1970s) led to the connection between TD learning and RL.

Combined and Fundamental Developments (1980s-1990s)

• Sutton & Barto (1980s) unified the threads of RL, focusing on concepts like TD learning, Q-Learning, and the Actor-Critic model.

• Q-Learning by Watkins (1989) and Werbos (1987) combined trial-and-error and dynamic programming, sparking further research in RL algorithms.

• REINFORCE (1992) by Ronald J. Williams is the first policy gradient method for optimizing the parameters of a policy directly instead of using value function like Q-learning

Modern Developments with Deep Neural Network (2010s – Now)

• AlphaGo from Google DeepMind beat a world champion in Go in 2016, using a combination of Monte Carlo Tree Search and deep learning, advancing RL significantly.

• Mnih et al. (2013) introduced the DQN algorithm, combining Deep Learning and Q-Learning, achieving human-level performance in Atari games.

• Several common RL algorithms have been boosted, such as TRPO (Trust Region Policy Optimization) (2015), PPO (Proximal Policy Optimization) (2017), and SAC (Soft Actor-Critic) (2018) contributing to the development of robot learning and other applications.

Recent Advances (2020s)

• Model-Based RL not only learn policies but also model their environment. This enables more efficient exploration and better sample efficiency. 

• Offline RL learning from pre-collected data rather than real-time interaction, is gaining traction in areas where real-world interaction is costly or impractical. This approach is particularly useful for learning from historical data without the need for exploration.

• Enhancing RL with Foundation Models: Recent research integrates RL with large pre-trained models like large language models (LLMs) and vision transformers (ViTs). These models provide rich, generalizable representations, enhancing decision-making in complex environments and improving the adaptability of RL agents.

• Post-Training Foundation Models with RL and Reinforcement Learning from Human Feedback (RLHF) is being used to fine-tune large pre-trained models after initial training. This post-training technique refines model behavior through human feedback and interaction, allowing for more robust and aligned models in tasks like natural language processing.

Challenges and the Future of Reinforcement Learning

While RL has shown remarkable progress, several challenges remain:

• Sample Efficiency: RL often requires vast amounts of data (interactions with the environment) to learn effectively, which can be both computationally expensive and time-consuming.

• Exploration vs. Exploitation: Striking the right balance between exploring new actions and exploiting known successful strategies is a major challenge, particularly when rewards are sparse or delayed.

• Generalization: RL agents sometimes struggle to generalize knowledge across different environments. An agent trained in a simulated world may not perform well in real-world conditions due to discrepancies between the two.

• Interpretability: Understanding how RL agents make decisions, especially in complex tasks, remains a challenge. Making RL models more interpretable is crucial for trust and transparency.

Video Explanation:

• The video is from CS 285 deep reinforcement learning course from Sergey Levine at UC Berkeley. In the lecture 2, part 1, the terminology and notation, definition of MDP and the reward functions/goal are introduced formally. This video can serve as a starting point for you to get comfortable with math in RL. (Runtime: 23 mins)

• The section of “What is Reinforcement Learning?” within the course of Unsupervised Learning, Recommenders, Reinforcement Learning under DeepLearning.AI is an intuitive introduction from Cousera, taught by Andrew Ng. It provides several demo videos to show how RL works on real hardware at an early stage with a brief description of the notations in RL. Note that RL has become mature within the last few years, so the demo and the status of RL from the video may be out of date. Still, this short introduction provides a concise and informative introduction of RL. (Runtime: 2 hrs 43 mins)
  
  Video Link: Unsupervised Learning, Recommenders, Reinforcement Learning by DeepLearning.AI on Coursera

Conclusion

Reinforcement learning is transforming the landscape of artificial intelligence by enabling machines to learn from experience and adapt their behavior over time. Its ability to tackle complex decision-making problems makes it an invaluable tool in fields ranging from gaming to healthcare to business.

As RL algorithms continue to improve in efficiency and scalability, we can expect even more exciting developments—smarter robots, more personalized healthcare, and breakthroughs in various industries. The future of reinforcement learning is bright, and we are just beginning to scratch the surface of its potential.

Related Articles

1. What does ‘policy’ in Reinforcement Learning mean?
2. What is Supervised Learning?
3. What is Unsupervised Learning?

Author Name: Hao-Lun Hsu, Duke University