Machine Learning Neural Network: Can Training Error and Testing Error be equal?

1
votes

Regressions learning rate: 0.001 Ratio of training to test data:50%

I trained my neural network and both training error and testing error started out at 0.120 then it both decreases steadily and train it until I reached 2,105 epochs. End results was both training error and training loss data was 0.006. Is this considered over-fitting, under-fitting, or did I made a huge error?. Also I would like to ask, if a good-fitting model has a low validation error(testing error) but still slightly higher then training loss then how far off are they from each other?. for example, 0.012 = validation loss(testing loss), 0.005 for training loss. Would a good fitting model look similar to that number?.

1
pythonmachine-learningneural-networkdeep-learningregression
Can you share the code? – datawrestler
It is unlikely but it can happen. Overfitting is when your validation error is much higher than your training error, underfitting is when both errors are high. You usually expect a low training error with a slightly higher validation error, so your results seems okayish. Try to compare the training and validation sets, maybe there could be a small mistake like printing the same error, or calculating the error with same sets. And for your last question yes, with that losses, it would look like a good fit. – umutto
@umutto so when we have two equal testing loss / training loss both at 0.006 is that over-fitting or under-fitting? – user6751157
@Teh_Hammer 0.006 is a good score so it is not underfitting. And since your test and training loss is very close to each other it is not overfitting. It looks like a good fit, but both losses being equal is not very likely (it could happen though) which means there could be an implementation error, probably during loss calculation or training / test set seperation. So if you double, triple check your model implementation and see no problem with it, it is not overfitting or underfitting, it is a good fit. – umutto
thank you @umutto I think it might have been my dataset is too little but I still get the same result after 4+ times of re-training and predicting. – user6751157

1 Answers

1
votes

For truly new testing data from a new source, one might expect the testing error to be higher than the training error because of technical source of variation or because of population bias (aka the batch effect).

For randomly split testing data used for the purpose of model validation, a much higher testing error is a sign of overfitting in my experience. However, there are actually no hard rules that says how much of a difference b/n testing and training is "overfit".

See this image from wikipedia: https://en.wikipedia.org/wiki/Overfitting#/media/File:Overfitting_svg.svg

You can see that simply minimizing the difference between training and testing error does not lead to the best model.

From your numbers specifically, a testing error of 0.012 is really good no matter what the training error is. You may be able to do better, but gains will start to become marginal when testing error is already that low.