Flatten matrix in R to four columns (indexes and upper/lower triangles)

Question

I'm using the cor.prob() function that's been posted several times around the mailing list to get a matrix of correlations (lower diagonal) and p-values (upper diagonals):

cor.prob <- function (X, dfr = nrow(X) - 2) {
  R <- cor(X)
  above <- row(R) < col(R)
  r2 <- R[above]^2
  Fstat <- r2 * dfr/(1 - r2)
  R[above] <- 1 - pf(Fstat, 1, dfr)
  R[row(R) == col(R)] <- NA
  R
}

d <- data.frame(x=1:5, y=c(10,16,8,60,80), z=c(10,9,12,2,1))

cor.prob(d)

> cor.prob(d)
           x           y           z
x         NA  0.04856042 0.107654038
y  0.8807155          NA 0.003523594
z -0.7953560 -0.97945703          NA

How would I collapse the above correlation matrix (with the correlations in the lower half, p-values in the upper half) into a four-column matrix: two indexes, the correlation, and the p-value? E.g.:

i  j   cor    pval
x  y   .88    .048
x  z  -.79    .107
y  z  -.97  0.0035

I've seen the answer to the previous question like this, but will only give me a 3-column matrix, not a four column matrix with separate columns for the p-value and correlation.

Any help is appreciated!

I just edited for clarity. I want to collapse the output from cor.prob() to a four-column table with two columns for the indices, one column for the correlations (which were in the lower diagonal of the original matrix), and one column for the p-values (which were in the upper half of the original matrix). — Stephen Turner

ChrisW ChrisW · Accepted Answer · 2012-08-24T20:38:36

well it's not a matrix, because you can't mix characters and numerics. But:

this is my first attempt (before your label swap):

m <- cor.prob(d)
ut <- upper.tri(m)
lt <- lower.tri(m)
d <- data.frame(i=rep(row.names(m),ncol(m))[as.vector(ut)],
            j=rep(colnames(m),each=nrow(m))[as.vector(ut)],
            cor=m[ut],
            p=m[lt])

now apply the correction I suggested below and you get

d <- data.frame(i=rep(row.names(m),ncol(m))[as.vector(ut)],
            j=rep(colnames(m),each=nrow(m))[as.vector(ut)],
            cor=m[ut],
            p=t(m)[ut])

finally your label swap, use row()/col(), and write it as a function:

f1 <- function(m) {
  ut <- upper.tri(m)
  data.frame(i = rownames(m)[row(m)[ut]],
            j = rownames(m)[col(m)[ut]],
            cor=t(m)[ut],
            p=tm[ut])
}

then

m<-matrix(1:25,5,dimnames=list(letters[1:5],letters[1:5])
> m
  a  b  c  d  e
a 1  6 11 16 21
b 2  7 12 17 22
c 3  8 13 18 23
d 4  9 14 19 24
e 5 10 15 20 25

> f1(m)
   i j cor  p
1  a b   6  2
2  a c  11  3
3  b c  12  8
4  a d  16  4
5  b d  17  9
6  c d  18 14
7  a e  21  5
8  b e  22 10
9  c e  23 15
10 d e  24 20

Can you explain what you expected if it wasn't this?

Flatten matrix in R to four columns (indexes and upper/lower triangles)

2 Answers