implement Gaussian distribution with singular sigma

Question

I am implementing the Gaussian distribution model on some data, if the sigma(covariance matrix) is singular, then it's not invertible, and will result in the failure in calculating the probability. I think add an Identity matrix to the sigma will make the sigma invertible, but that will make the model not fits the data.

Is there a way to make the sigma matrix invertible and keep the model fitting data?

Have a set of data: (x1, x2)_1 , (x1, x2)_2 , ... , (x1, x2)_i . where x1 and x2 are continues real numbers and some (x1, x2) can appear serval times, And I assumpt that those data follow Guassian distribution, and then can calculate the mean vector as (mean(x1), mean(x2)), and then calculate the covariance matrix as usual. And in some case the covariance matrix may be singular, I think add some random small shifts to it can make it nonsingular, but I don't know how to do it correctly so that the model can still fit data well.

Answer 1

You only need to model one dimension of the data with a 1D gaussian distribution in this case.

If you have two-dimensional data {(x1,x2)_i} whose covariance matrix is singular, this means that the data lies along a straight line. The {x2} data is a deterministic function of the {x1} data, so you only need to model the {x1} data randomly. The {x2} data follows immediately from {x1} and is no longer random once you know {x1}.

Here is my reasoning:

The covariance matrix would look something like this, since all covariance matrices are symmetric:

| a  b |
| b  c |

Where a = var(x1), c = var(x2), b = cov(x1,x2).

Now if this matrix is singular, the second column vector would have to be a scalar multiple of the first (since they are linearly dependent). Let's say the constant is k. Then:

b = k*c
a = k*b = k*k*c

So the covariance matrix really looks like:

| k*k*c  k*c |
|  k*c    c  |

Here there is only one parameter c = var(x2) which determines the distribution (since k can be anything), so the data in inherently one-dimension. Modelling it with one variable x1 is enough. Another way of seeing this is by checking that the Pearson Correlation Coefficient for this distribution is b/(sqrt(a)*sqrt(c)) = 1.