I'm implementing continuous reinforcement learning (Continuous Control with Deep Reinforcement Learning https://arxiv.org/abs/1509.02971) but I have ran into problems when optimizing the policy-neural network.
As recommended in the paper I maximize policy's Q-value: max Q(state,action=policy(state)) by calculating and following the gradient (gradient ascent). However, because the Q-neural network is not (initially) perfect approximation of Q(state,action) and becomes very large/infinite when policy's action becomes large/infinite I ran into problems. Q->infinite when action->infinite and this means policy's weights are optimized to extremely large or infinite values. Morover, when policy starts outputting near infinite values this means Q-values become too large and reinforcement values becomes irrelevant as "r + gamma*Q" is dominated by extremely large Q-values.
What is your recommended solutions for this? I'm planning to restrict policy's outputs to [0,1] by using sigmoidal output layer instead of linear one but this seems to seriously hamper the optimization process and the algorithm cannot learn good policies anymore..
