RMSE and MAE of predicted values is better than benchmark, but predicted values are only closer to actual values half the time

Question

I'm trying to learn some machine learning basics to predict football players' fantasy football points.

I've built a dataset and done some modelling. My plan was to compare my predictions to projections made by fantasy football experts.

For my test data, the mean absolute error of my predictions = 66.2, while the mean absolute error of the experts' projections = 109.3. The RMSE of my predictions on the test set = 90.2, while the experts' projections have an RMSE of 150.1.

I thought this would mean that my model's projections would beat the expert projections consistently. However, this was not the case. When I look at the specific individuals in my test set, my model's predictions only beat the experts' predictions 56% of the time.

Does anyone have any ideas for why this might be? If so, what could I do to try to address the issue?

(One thing I noticed is that my model doesn't do well at projecting the highest scoring players. It was worse than the experts' on all of the top 5, though it did great in the middle tiers.)

Answer 1

It could be that your model is overfitting on the data. That will make it look like your model is great during testing, but it won't be accurate to new data in real life.