Motivation
LunarLander is a classic control problem: balance thrust to land softly between two flags. It's a perfect testbed for deep reinforcement learning — sparse rewards, continuous state, and the need to balance exploration against exploitation. I wanted to implement DQN from the ground up and actually solve it.
Architecture
The agent observes the environment state, picks actions via an ε-greedy policy, and learns from sampled past transitions stored in a replay buffer. A separate target network stabilizes the learning signal.
🌒Gym Envstate, reward
→
🎲ε-greedy Policyexplore/exploit
→
🗃️Replay Buffersample batches
→
🧠DQN + Target Netstable updates
Experience replay + a target network are what make DQN converge.
What I built
- An RL agent that lands a spacecraft, optimizing control for a stable touchdown.
- A Q-learning agent in OpenAI Gym reaching 95% convergence in 800 episodes with an ε-greedy strategy.
- Performance pushed to a reward score of 500 using experience replay and target networks.
- A full DQN implementation tuned for the exploration–exploitation balance.
Results
500Reward score
95%Convergence
800Episodes to converge
Challenges
- Training instability: naive Q-learning diverged — a target network was needed to stabilize the bootstrapped targets.
- Exploration: tuning the ε-decay schedule balanced early exploration against late exploitation.
- Sample efficiency: experience replay broke correlation between consecutive frames and sped up learning.
Learnings
- Stability tricks are the whole game in deep RL — replay and target networks separate "diverges" from "solves it".
- Reward curves are noisy; judging progress needs moving averages, not single episodes.
- Hyperparameters (ε-decay, learning rate, buffer size) dominate outcomes.