For handling configuration in Release Management, there are two techniques generally used Config Per Environment and Tokenization.
If you prefer a clean separation between build and deploy. To achieve that, recommend tokenizing configuration.
As you mentioned above, suggest you to use Config Per Environment.
In “config per environment”, you essentially hard-code a config
file per environment. At deploy time, you overwrite the target
environment config with the config from source control. This could be
an xcopy operation, but hopefully something a bit more intelligent
– like an ARM Template param.json file. When you define an ARM
template, you define parameters that are passed to the template when
it is executed. You can also then define a param.json file that
supplies the parameters.
More details please take a look at this wonderful blog: Config Per Environment vs Tokenization in Release Management