Linear basis in scikit-learn Gaussian Process

Question

I am rewriting a code in MATLAB in Python. The code performs Gaussian Process regression using fitrgp. In Python, I am using scikit-learn and its GaussianProcessRegressor. In both cases, the GP learns the mapping betwen input and output. The core of my MATLAB and Python codes are as follows:

-MATLAB

prediction_train=[];

%% z-score of input
[normalized_input_train,mu_train,std_train]=zscore(input_data);

parfor j=1:1:size(output_data,2)
j %prints time component

z1=squeeze(output_data(:,j));

beta1=fitrgp(normalized_input_train,z1,'BasisFunction','linear','KernelFunction','ardmatern32'); 

beta1=compact(beta1);

prediction_tmp=predict(beta1,normalized_input_train);

prediction_train=[prediction_train, prediction_tmp];

end

which trains the GP on some training input_data, and learns the prediction for the same input_data, to be later compared with output_data. This script implements a linear basis with ard Matern 3/2 kernel function.

-Python

# Instantiate a Gaussian Process model
kernel = 1.0 * Matern(nu=1.5)
gp = GaussianProcessRegressor(kernel=kernel)

# z-score of input
scale = StandardScaler()
normalized_input_train = scale.fit_transform(input_data)

prediction_train = np.zeros((np.shape(output_data)))

for i in range(np.shape(output_data)[1]):
    print("time component", i)

    gp.fit(normalized_input_train, output_data[:,i])

    prediction_tmp, sigma = gp.predict(normalized_input_train, return_std=True)

    prediction_train[:,i] = prediction_tmp

This should do exactly the same as my MATLAB code. However, I am not sure how to implement the linear basis option that is implemented in the MATLAB script.

jejjohnson jejjohnson · Accepted Answer · 2019-10-29T20:29:20

So I'm not familiar with all of the workings of the MATLAB implementation but after browsing through the documentation I believe I understand it as the 'mean function' for the Gaussian Process regression algorithm. The implementation on scikit-learn assumes a zero mean function and there is no direct way of specifying your own custom mean function for that implementation.

If it is absolutely necessary that you have the mean function then I recommend you try the GPy library which has more customization including the ability to specify a mean function out of the box. Here is an example. The learning curve is a bit higher for the GPy package but if you want more custom options then I find this package to be better than the one in scikit-learn.

On a side note, I noticed in your MATLAB implementation you specified the ardmatern kernel. But in your sklearn implementation you set the nu parameter to 1.5 but you did not initialize the length scale to be the length of your features (i.e. the number of dimensions for your input x).

e.g.

kernel = 1.0 * Matern(
   nu=1.5, 
   length_scale=np.ones(normalized_input_train.shape[1]),
)

Be sure to do so otherwise those two kernels are not equivalent as the term 'ARD' means that there is a length scale for every feature in your input. See the length_scale parameter for the Matern kernel in the sklearn docs for more details.

Hope this helps.

Linear basis in scikit-learn Gaussian Process

1 Answers