calculate root-mean-squared-error of regression in bouts

Question

lets assume that I get the following pandas dataframe for my regression analysis.

import pandas
import math
import numpy

df = pandas.DataFrame(numpy.random.randint(0,100,size=(100, 2)), columns=['labels','predictions'])

I would like now to calculate the RMSE as

math.sqrt(numpy.mean((df["predictions"] - df["lables"]) ** 2))

for values of the labels in interval of 7

Hereby a very ugly code that does the job...it would be nice if you help me to pythonize it...

# define step
step = 7
# initialize counter
idx = 0
# initialize empty dataframe
rmse = pandas.DataFrame(columns=['bout' , 'rmse'],index=range(0,len(range(int(df['labels'].min())+step,int(df['labels'].max()),step))))

# start loop to calculate rmse every 7 units
for i in range(int(df['labels'].min())+step,int(df['labels'].max()),step):

    # select values in interval
    df_bout = df[(df['labels']>=i-step) & (df['labels']<i)]

    # calculate rmse in interval
    rmse.loc[idx] = [str(i-step)+'-'+str(i),math.sqrt(numpy.mean((df_bout.predictions - df_bout.labels) ** 2))]

    # increment counter
    idx = idx + 1

Lukas Thaler Lukas Thaler · Accepted Answer · 2018-10-23T08:46:01

I apologize for my misunderstanding in the beginning. The following code snippet gives the results you wish

from sklearn.metrics import mean_squared_error
import pandas
import math
import numpy

df = pandas.DataFrame(numpy.random.randint(0, 100, size = (100, 2)), columns = ['labels','predictions']).sort_values(by = 'labels', ascending = True)
def rmse(df):
    return numpy.sqrt(mean_squared_error(df['labels'], df['predictions']))

output = df.groupby(numpy.floor(numpy.array(df['labels'] / 7))).apply(rmse)
rmse_df = pandas.DataFrame({'bout': [str(int(output.index[i] * 7)) + ' - ' + str(int(output.index[i + 1] * 7)) for i in range(output.shape[0] - 1)], 'rmse': output.values[:-1]})

You can change the 7s in my code for your variable step if you want to dynamically change the step size

calculate root-mean-squared-error of regression in bouts

1 Answers