Debugging Reinforcement Learning Model (MsPacman)

Question

I'm new to RL and I'm attempting to train an RL agent to play MsPacman in PyTorch. I've adapted the code from this tutorial on the PyTorch page for my problem. The DQN has the following architecture:

DQN(
  (conv1): Conv2d(1, 32, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
  (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (linear1): Linear(in_features=7040, out_features=512, bias=True)
  (linear2): Linear(in_features=512, out_features=9, bias=True)
)

I'm using the actor (policy) and critic (target) method with replay memory which has the following settings:

Replay Buffer: 100,000
Target Update: Every 10,000 steps
Bath Size: 128
Discount Rate: 0.999

For the exploration trade-off I'm using epsilon-greedy with the following curve:

where the x-axis is the step number (in million) and y the probability of selecting a random action.

The update of the policy network looks like this:

# next_state_values = Q-values precited by the target network
# GAMMA = discount rate (0.999)
# reward_batch = rewards for the states
expected_state_action_values = (next_state_values * GAMMA) + reward_batch

Loss calculation:

# state_action_values = actions taken by the policy agent
# expected_state_action_values - this is calculated above
loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

Updating the policy (gradients are clamped):

optimiser.zero_grad()
loss.backward()
for param in policy_net.parameters():
    param.grad.data.clamp_(-1, 1)
optimiser.step()

While training the agent I plot the duration of each episode shown below (the orange line shows the average of the previous 100 episodes):

After 4,000 episodes the agent isn't really progressing and gets stuck in corners like below:

Any idea what could be the issue? Some tips and pointers would be extremely helpful.

dilaudid dilaudid · Accepted Answer · 2020-06-20T15:39:34

It would be interesting to see the episode rewards, as that is what the agent is optimising for. Is the agent getting stuck in corners from the beginning? If so, I would guess that the epsilon greedy policy (which is pretty much pure random at the beginning) is not implemented properly. I assume that the agent gets rewards for eating the dots, so it should be motivated to move around.

Try a pure random agent and see how that progresses, you can plot that as a baseline which will tell you if your agent is learning anything at all.

Debugging Reinforcement Learning Model (MsPacman)

1 Answers