All challenge writeups

The Maze Runner

Explore reward shaping and policy gaps in a DQN maze agent to reach a forbidden state.

You think running a maze is hard? How about finding the exit inside you.

Overview

You’re given a pre-trained reinforcement learning (RL) agent (a DQN) that navigates a 12×12 maze. Under normal play, it moves from the start (0,0) to the goal (11,11) while avoiding a forbidden state at (6,6).

Your mission: exploit flaws in the agent and/or environment to force the agent into (6,6), which reveals the flag.

  • Challenge Type: RL exploitation / reward hacking
  • Difficulty: Hard
  • Flag: IEEE{rl_reward_hacking_forbidden_6_6}

Phase 1 — Understand the Maze Environment

What to do (explanation only)

  • State space: 12×12 grid (144 positions).
  • Action space: 4 moves (Up=0, Right=1, Down=2, Left=3).
  • Rewards:
    • 0.1 per step
    • +100 when reaching the goal (11,11)
    • Special quirks: “bonus” rewards when certain hidden sequences occur.
  • Forbidden state: (6,6) contains the flag, but the agent avoids it under normal policy.

Key observations

  1. Reward shaping flaws → Circular movement patterns yield unintended cumulative rewards.
  2. Policy gaps → The agent was undertrained near (6,6).
  3. Environment exploits → Specific action sequences let you phase through walls.

Phase 2 — Analyze Agent Behaviour

What to do (explanation only)

  • Run the demo (python maze_agent_demo.py) and watch normal navigation.
  • Use policy visualization to see action preferences at each state.
  • Notice how the agent consistently avoids the central forbidden zone.
  • Look for states where Q-values are unstable → training blind spots.

Phase 3 — Develop Exploits

1. Reward Hacking (circular moves)

  • Repeated sequences (e.g., Up, Right, Down, Left) trick the reward function.
  • The agent accrues enough “neutral” reward to push into risky states.

2. Phase-through sequence

  • Some sequences (e.g., rapid alternating Left/Right at walls) bypass obstacles.
  • This lets you guide the agent toward (6,6) even if the direct path is blocked.

3. Direct navigation into forbidden state

  • Once close to (6,6), craft a precise sequence that exploits policy gaps.
  • Because the agent lacks experience here, adversarial inputs can steer it inside.

Code snippet — guided sequence execution

from forbidden_maze_env import ForbiddenMazeEnv
from dqn_agent import DQNAgent
import torch

# Load environment and agent
env = ForbiddenMazeEnv()
agent = DQNAgent(state_size=144, action_size=4, hidden_size=128)
agent.qnetwork_local.load_state_dict(torch.load("vulnerable_agent.pth"))
agent.qnetwork_local.eval()

# Force agent with crafted sequence
actions = [1,1,2,2,3,0,0,2]  # Example: Right, Right, Down, Down, ...
state, _ = env.reset()
for a in actions:
    state, reward, done, _, _ = env.step(a)
    env.render()
    if done:
        break

Phase 4 — Reach the Forbidden State

Success condition

When the crafted exploit guides the agent into (6,6), the environment prints:

🚩 FORBIDDEN STATE REACHED! 🚩
Flag: IEEE{rl_reward_hacking_forbidden_6_6}

Original Notion writeup