Unable to create covariance matrix from random vector in Matlab

Question

I'm trying to build a Gaussian Mixture Model using random initializations and compare the results with one using Kmeans initializations. However, I have difficulty creating the initial covariance matrix. I randomly selected 10 data points from my data set of 2500 data points (each "point" is actually an image), and used them as the means. Then I'm trying to create the covariance matrix from each of these random points.

Here's what I have.

% Randomly initialize GMM parameters
rng(1);
rand_index = randperm(2500);
Mu = data(:,rand_index(1:10));
for i = 1 : 10
    Sigma(:,:,i) = cov(Mu);
    Pxi(:,i) = mvnpdf(data', Mu(:,i)', Sigma(:,:,i));
end 

data is a 50x2500 matrix. I keep getting an error because my Sigma is of the wrong size, or it's not positive definite, etc.

For example, the code above gave the error

Error using mvnpdf (line 116)
SIGMA must be a square matrix with size equal to the number of columns in X, or a row vector with length equal to the number of
columns in X.

If I use

Sigma(:,:,i) = cov([Mu(:,i) Mu(:,i)]');

I get the error

Error using mvnpdf (line 129)
SIGMA must be a square, symmetric, positive definite matrix.

How should I create this covariance matrix?

Answer 1

I assume what you are experiencing is not happening at every run. This is a numerical instability that you can avoid using a simple technique:

%Add a tiny variance to avoid numerical instability
Sigma(:,:,i) = cov([Mu(:,i) Mu(:,i)]');
D = size(Sigma,1);
Sigma(:,:,i) = Sigma(:,:,i) + 1E-5.*diag(ones(D,1));