Neural networks fails to approximate simple multiplication and division

Question

I am trying to fit simple feedforward neural networks on simple data where my goal is to just approximate (abc)/d

max_a=2
max_b = 3000
max_c=10
max_d=1

def generate_data(no_elements=10000):
    a = np.random.uniform(0,max_a,no_elements)
    b = np.random.uniform(1,max_b,no_elements)
    c=np.random.uniform(0.001,max_c,no_elements)
    d=np.random.uniform(0.00001,max_d,no_elements)
    df=pd.DataFrame({"a":a,"b":b,"c":c,"d":d})

    e=(df.a*df.b*df.c)/df.d
    df["e"]=e
    return(df)

this is how i am generating data

then I did data normalization

df = generate_data(5000)
np_df=df.iloc[:,:4]
means=np.mean(np_df,axis=0,keepdims=True)
stds=np.std(np_df,axis=0,keepdims=True)
x_train = (np_df-means)/stds
y_train = np_df[:,4]

and I have built a simple pytorch network for regression so as it has to predict 'e'

class network_Regression(nn.Module):
    def __init__(self,layers):
        super(network_Regression, self).__init__()
        self.linear = nn.ModuleList()
        self.relu = nn.ModuleList()
        self.layers = layers
        for i in range(len(layers)-1):
            self.linear.append(nn.Linear(layers[i],layers[i+1]))
            if i+1 !=  len(layers)-1:
                self.relu.append(nn.ReLU())

    def forward(self,out):
        for i in range(len(self.relu)):
            out = self.linear[i](out)
            out = self.relu[i](out)
        out = self.linear[-1](out)
        return out


model = network_Regression([4,10,10,1])
criterion= nn.MSELoss()
optimizer=optim.Adam(model.parameters())

but when I tried to train these networks tried epochs from [1000 to 0.5M]

still, it isn't able to find a simple formula ((abc)/d)=e

I tried to change various hidden layer levels but loss been around 9 digits

model.train()
num_epochs = 1000
loss_list=[]
for epoch in range(num_epochs):
    for batch_idx, (data, target) in enumerate(data_loader):
    #print(batch_idx)
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data.float())
        loss = criterion(output, target.float())
        #print(batch_idx, loss.data[0])
        loss.backward()
        optimizer.step()
        if epoch >2:
            if batch_idx % 200 == 0:
                loss_list.append(loss.data.item())
        if batch_idx % 400 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, batch_idx * len(data), len(data_loader.dataset),
            100. * batch_idx / len(data_loader), loss.data.item()))

Possible duplicate of PyTorch will not fit straight line to two data points — Berriel
doesn't look like duplicate output contains various values but issue is MSE which is very high array([[104036.31], [143887. ], [121395.36], ..., [238713.61], [114323.66], [153722.58]], dtype=float32) — ganesh

ganesh ganesh · Accepted Answer · 2019-04-22T06:37:14

Looks like neural networks are bad in multiplication and division.

check out this for details.

So basically I have to log transform my data, in the above case to approximate ((abc)/d)=e neural network has to figure out simple addition and subtraction.As per this question complicated multiplication and division becomes ln(a) + ln(b)+ln(c)-ln(d) =ln(e) and just after take inverse log of ln(e) this idea works well.

Neural networks fails to approximate simple multiplication and division

1 Answers