How to get a pariwise correlation dataframe from a matrix?

Question

I have a matrix on which I compute the correlations between my columns. I created a function which gives the result in the form of correlation matrix (because of the apply() function), but I would like to get a pairwise correlation dataframe directly. Is it possible to do that withouth any intermediate matrix : matrix -> pairwise correlation dataframe

cor_rho<-function(y) {
res <- foreach(i = seq_len(ncol(y)),
.combine = rbind,
.multicombine = TRUE,
.inorder = FALSE,
.packages = c('data.table', 'doParallel')) %dopar% {
apply(y, 2, function(x) 1 - ((var(y[,i] - x)) / (var(y[,i]) + var(x))))}
return(res)}

This is the same function, I just added some lines to get the data.frame I want

cor_rho<-function(y) {
res <- foreach(i = seq_len(ncol(y)),
.combine = rbind,
.multicombine = TRUE,
.inorder = FALSE,
.packages = c('data.table', 'doParallel')) %dopar% {
apply(y, 2, function(x) 1 - ((var(y[,i] - x)) / (var(y[,i]) + var(x))))}
colnames(res)=rownames(res)=colnames(y)
Df<-data.frame(var1=rownames(res)[row(res)[upper.tri(res)]],
var2=colnames(res)[col(res)[upper.tri(res)]],
corr=res[upper.tri(res)])
return(Df)}

which gives me something like this

var1 var2 value
var1 var3 value
var2 var3 value

Rémi Coulaud Rémi Coulaud · Accepted Answer · 2019-05-08T20:39:12

I use only dplyr and reshape2, is it working for what you want ?

library(reshape2)
library(dplyr)

set.seed(1)
n <- 10
df <- data.frame(var1 = rnorm(n), var2 = rnorm(n), var3 = rnorm(n))
melt(cor(df), id = c("Var1", "Var2")) %>%
  filter(Var1 != Var2) %>%
  filter(duplicated(value))

I made an assumption which can be anoying for you, it is to suppose no duplicate between correlation coefficient neither for symetric relation. I hope it will help.

Edit 1 :

I look on internet, I find the following package : corrr in which their is a function correlate which return you directly a data.frame. If you want the specify output, the following lines gives you something near :

install.packages('corrr')
library(corrr)
stretch(correlate(df))

But after that you need to do the same trick as above to have only what you want.

Edit 2 :

I look again on internet to see what could help you, in fact the only package which take a matrix in input and gives a data.frame in output is corrr but I tested it on a bigger matrix and it is realy slow.

n <- 100
p <- 8000
m <- n * p 
df <- data.frame(matrix(rnorm(m), nrow = n, ncol = p))


t <- Sys.time()
m_cor <- cor(df)
t <- Sys.time() - t
t1 <- t 

library(corrr)

t <- Sys.time()
m_cor <- correlate(df)
t <- Sys.time() - t
t2 <- t

library(propagate)

t <- Sys.time()
m_cor <- bigcor(df)
t <- Sys.time() - t
t3 <- t

Wherease propogate seems to be a good thing to deal with very large matrix but I'm not sure to well understand the type ff of the output. In my exemple, the basic cor function is faster than the two others solution. Do you find any solution ?

I will be happy to see your proposal because it is an interesting question.

How to get a pariwise correlation dataframe from a matrix?

1 Answers