Creating a correlation matrix from a corpus in R

Question

I have a large corpus and I would like to create a correlation matrix for all the terms in the entire corpus. I can find correlations for any given word in the corpus using the following code:

      findAssocs(corp_dtm, terms = "serachword", corlimit = 0.01)

But I would like to plot this data, using the correlations as weights, so I need a matrix with all the correlations. Is there an easy way to do this?

     hello   world   my     name     is     liam

hello   1      .3     .04    .21     .88    .00

world   .3     1

my      .04            1

name    .21                   1

is      .88                           1

liam    .00                                    1

Like this, but all filled in.

Thanks!

It's easier to help you if you include a simple reproducible example with sample input and desired output that can be used to test and verify possible solutions. — MrFlick

phiver phiver · Accepted Answer · 2019-11-25T17:25:34

As far as I know, there are no correlation functions for sparse matrices. So you need to transform the sparse matrix into a normal matrix first. See line of code below.

But I advice against this as this will first create a dense matrix and you have a good chance of running into memory issues if your document term matrix is even slightly large. And you indicated this is the case.

cor_matrix <- cor(as.matrix(corp_dtm))

Creating a correlation matrix from a corpus in R

1 Answers