Find correlation coefficients between two big data frames

Question

I have to big data frames for genomic analysis. Each of these data frames has 500k rows and 100 columns. Those 100 columns represent measurements for each gene. What I want to do is calculating the Spearman's correlation coefficient for each gene across all 100 values in both data farms. Example:

df1
genename      x1      x2     x3  ..............x100
gene1      0.236   0.589  0.896               0.789
gene2      -0.361  0.782  0.583               0.478


df2
genename      x1      x2     x3  ...............x100
gene1      0.101   0.256  0.026               0.0.56
gene2      -0.231  0.569  0.158               0.0223

What I want here for example is to find all correlation coefficients for gene1 across all 100 obs. That means I should have 100 correlation coff for each gene

Please explain the sentence: "That means I should have 100 correlation coff for each gene" — S Rivero
I am not sure if I understand you correctly. But spearman correlation is a nonparametric measure of rank correlation, so you can not compare x1_df1 vs x1_df2(1 value vs 1 value), actually you can but the corr value is going to be 1. What you can do is to get 1 corr value per gene, in other words, compare x1,x2,x3,...x100 in df1 vs x1,x2,x3,...x100 in df2. — S Rivero
Understood. You want to calculate the correlation in a matrix. The correlation values can be calculated with cor, however the p-values it is more complicated. For that you can use other functions like rcorr (Hmisc package) or corr.test ( psych package). Some information: sthda.com/english/wiki/… — S Rivero

user387832 user387832 · Accepted Answer · 2018-05-24T15:13:25

The phrasing "I will have 100 correlation coefficients for each gene" is confusing to me given each row is labelled as geneX and your description makes it sound like you are trying to find the correlation coefficient between xX in df1 and xX in df2, where the rows would be your observations. Assuming you are trying to find the Spearman correlation between the similarly labelled rows of df1 and df2 (i.e., the correlation between gene1 in df1 and df2), it can be done as such:

m1 <- as.matrix(df1)
m2 <- as.matrix(df2)

res <- c()

for (i in 1:nrow(m1)) {
    res <- c(res, cor(m1[i,], m2[i,], method = "spearman")))
}

This might take a bit of time, given the size of the data.frames. It takes about 20 secs to do 100000 row matrices on my machine. It might be worth looking into mclapply if you have access to multiple cores.

Find correlation coefficients between two big data frames

1 Answers