dimensionality reduction for non square matrix?

Question

Im going to do dimensionality reduction by using PCA/SVD for my extracted features.

Suppose if I want to do classification using SIFT as the features and SVM as the classifier.

I have 3 images for training and I arrange them in a different row..

So 1st row for 1st images, 2nd row for second images and 3rd row for the 3rd image... And the columns represents the features

   A=[ 1     2     3     4     5
       4     5     6     7     8
       0     0     1     9     0]

To do dimensionality reduction (from my 3x5 matrix/non square matrix), we have to do A*EigenVector

Now I have to extract the eigen vector from my A matrix And from what I understand, that SVD is for non square matrix, so to perform PCA (eigs function), I need to make it square by multiplying it with it's transpose)

Here is if I do SVD directly

      [u1,s1,v1] = svd(A);

I got

u1 =

   -0.4369    0.1426    0.8882
   -0.8159    0.3530   -0.4580
   -0.3788   -0.9247   -0.0379


v1 =

   -0.2229    0.2206   -0.7088   -0.6070   -0.1754
   -0.2984    0.2910   -0.3857    0.4705    0.6754
   -0.3966    0.2301   -0.0910    0.5382   -0.7012
   -0.6547   -0.7495    0.0045   -0.0598    0.0779
   -0.5248    0.5020    0.5836   -0.3419    0.1233

and when I use PCA (eigs function) {as I arrange the feat of images in different row, so I need to do A*A'}, I got

c=A*A'
 [e1 e2]=eigs(c);

e1 =

    0.4369    0.1426    0.8882
    0.8159    0.3530   -0.4580
    0.3788   -0.9247   -0.0379

My question is:

is that right that I used it in SVD or in the PCA (by converting t into A*A' matrix) will give me he same eig vectoe (e1 and u1)??
As I arrange my images in different rows and the features for each images in different column. and PCA/SVD is suing to extract the eig vector which represent the relation between the variable.. So in this case the variable would be the row (images) or the columns (features)??
Do I have to convert my matrix into covariance matrix by using (Cov function) if I want to use eigs function?? Or it will be done by eigs function manually??

I do really appriciate any answer

Dariush Jafari Dariush Jafari · Accepted Answer · 2014-05-24T07:04:08

Suppose you have n-dimension samples and you want to reduce it to d-dimension data by PCA.
Suppose your data are stored in matrix A_nxN (N is the number of samples(here images)).
here n=3 and N=5.
We define a correlation matrix R = A*A' (nxn). You can use the covariance matrix instead.
Calculate the eignen vectors of R and corresponding eigen values:

R = A*A';
[eigVec, eigVal] = eig(R)

eigVec =

 0.8882    0.1426    0.4369
-0.4580    0.3530    0.8159
-0.0379   -0.9247    0.3788

eigVal =

1.7728         0         0
     0   49.6457         0
     0         0  275.5815

Note that columns of eigVec are the eigen vectors of R. Some of the eigen values will be zero or if not, you can take a threshold. So you can eliminate the corresponding eigen vectors:

T = eigVec(:, 2:3)
T =

 0.1426    0.4369
 0.3530    0.8159
-0.9247    0.3788

now T is a nxd matrix.
For any row vector X_1xn the X_1xn*T_nxd will result a Y_1xd output.
The final answer;

B = A'*T;

dimensionality reduction for non square matrix?

1 Answers