Why do we need to pass the gradient parameter to the backward function in PyTorch?

Question

According to the docs, when we call the backward function to the tensor if the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying gradient.

import torch
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([1.,1.])) 

print(a.grad)

Output: tensor([20., 20.])

Now scaling the external gradient:

a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([2.,2.])) #modified

print(a.grad)

Output: tensor([40., 40.])

So, passing the gradient argument to backward seems to scale the gradients.
Also, by default F.backward() is F.backward(gradient=torch.Tensor([1.]))

Apart from scaling the grad value how does the gradient parameter passed to the backward function helps to compute the derivatives when we have a non-scalar tensor?
Why can't PyTorch calculate the derivative implicitly without asking explicit gradient parameter as it did for the scalar tensor?

kHarshit kHarshit · Accepted Answer · 2021-03-26T05:48:58

It's because PyTorch is calculating the jacobian product. In case of scalar value, .backward() w/o parameters is equivalent to .backward(torch.tensor(1.0)).

That's why you need to provide the tensor with which you want to calculate the product. Read more about automatic differentiation.

Why do we need to pass the gradient parameter to the backward function in PyTorch?

1 Answers