Passing list of lists as input to scipy.optimize.curve_fit

Question

I am trying to solve an example marketing mix model problem using python and the curve_fit function.

I need to fit two sets of parameters, which i add to my function as a * arg list of lists. I can get the curve fit to work for one set of parameter (a single list) but not two.

Code

#import packages
from scipy.optimize import curve_fit
import pandas as pd
import numpy as np
from statsmodels.tsa.filters.filtertools import recursive_filter as rec

Dataset

a = np.array(0).repeat(150)
b = np.array(0).repeat(150)
c = np.array(0).repeat(150)
a[0:90] = np.random.uniform(5,10,(90,))
b[50:150] = np.random.uniform(20,40,(100,))
c[30:100] = np.random.uniform(5,25,(70,))
df = pd.DataFrame({'a':a,'b':b,'c':c})

Fitting one set of parameters

def mmm(data,*param):
    dic = {}
    j = 0
    for i in data:
        dic[i] = rec(data[i],param[j])
        j += 1
    return(np.sum(pd.DataFrame(dic),1))

The function, applies the recursive filter to each field in the data argument with a different lambda parameter and returns the dataframe row sum.

kpi = mmm(df,*(0.5,0.5,0.1)) + np.random.uniform(-5,5)

When passing a *argument to the scipy curve fit function you have to define a function that outputs a function. As described here:Pass tuple as input argument for scipy.optimize.curve_fit

Fit the curve

a = np.zeros(3)
def make_func():
    def mmm(data,*param):
        dic = {}
        j = 0
        for i in data:
            dic[i] = rec(data[i],param[j])
            j += 1
        return(np.sum(pd.DataFrame(dic),1))
    return(mmm) 
leastsq, covar = curve_fit(make_func(),df,kpi,a)

print(leastsq)
array([0.87560795, 0.87192766, 0.84864161])

Fitting two sets of parameters

def mmm(x,*arg):
    c = args[0]
    a = args[1]
    dic = {}
    j = 0
    for i in x:
        dic[i] = c[j] * rec(x[i], a[j])
        j += 1
    return(np.sum(pd.DataFrame(dic),1))   

The function, applies the recursive filter to each field in the data argument with a different lambda (a), multiplies it by a scalar (c) and takes the row sum of the dataframe.

args = [[4,5,3],[0.2,0.4,0.5]]
kpi = mmm(df,*args) + np.random.uniform(-5,5)

Fit the curve

args = np.zeros(6)
def make_func():
    def mmm(x,*args):
        c = args[0]
        a = args[1]
        dic = {}
        j = 0
        for i in x:
            dic[i] = c[j] * rec( x[i], a[j])
            j += 1
        return(np.sum(pd.DataFrame(dic),1))
    return(make_func)
leastsq, covar = curve_fit(make_func,df, kpi, p0=args)

Using the same method as for one list of parameters spits out an error for two. The error is as follows:

TypeError: make_func() takes 0 positional arguments but 7 were given

Is there something else, i have to do in order to get this code working?

Cheers,

Answer 1

There are two things which looks to me the source of error.

1) In the last part where you fit the curve in the function make_func(), you are returning the function itself. If I compare it to the previous function definition, I think it should be return(mmm).

2) args = np.zeros(6) results in an array of zeros which you pass as an argument to make_func(). You then assign c = args[0] and a = args[1] so basically c=0 and a=0 which are scalar variables. Now in the mmm(x,*args): function you use dic[i] = c[j] * rec( x[i], a[j]). Here pops up the IndexError: invalid index to scalar variable. because a and c are scalars but you are using index operations on them.