R: errors in cor() and corrplot()

Question

Another stumbling block. I have a large set of data (called "brightly") with about ~180k rows and 165 columns. I am trying to create a correlation matrix of these columns in R.

Several problems have arisen, none of which I can resolve with the suggestions proposed on this site and others.

First, how I created the data set: I saved it as a CSV file from Excel. My understanding is that CSV should remove any formatting, such that anything that is a number should be read as a number by R. I loaded it with

brightly = read.csv("brightly.csv", header=TRUE)

But I kept getting "'x' must be numeric" error messages every time I ran cor(brightly), so I replaced all the NAs with 0s. (This may be altering my data, but I think it will be all right--anything that's "NA" is effectively 0, either for the continuous or dummy variables.)

Now I am no longer getting the error message about text. But any time I run cor()--either on all of the variables simultaneously or combinations of the variables--I get "Warning message: In cor(brightly$PPV, brightly, use = "complete") : the standard deviation is zero"

I am also having some of the correlations of that one variable with others show up as "NA." I have ensured that no cell in the data is "NA," so I do not know why I am getting "NA" values for the correlations.

I also tried both of the following to make REALLY sure I wasn't including any NA values:

cor(brightly$PPV, brightly, use = "pairwise.complete.obs")

and

cor(brightly$PPV,brightly,use="complete")

But I still get warnings about the SD being zero, and I still get the NAs.

Any insights as to why this might be happening?

Finally, when I try to do corrplot to show the results of the correlations, I do the following:

brightly2 <- cor(brightly) Warning message: In cor(brightly) : the standard deviation is zero corrplot(brightly2, method = "number") Error in if (min(corr) < -1 - .Machine$double.eps || max(corr) > 1 + .Machine$double.eps) { : missing value where TRUE/FALSE needed

And instead of making my nice color-coded correlation matrix, I get this. I have yet to find an explanation of what that means.

Any help would be HUGELY appreciated! Thanks very much!!

read in your data as brightly and paste into your question the output of str(brightly), both with the NAs and without. — jeremycg
Just to clarify--everywhere where I've got "brightly" above, you want me to instead use "str(brightly)," run the same code in R, and paste the output here? Can definitely do, just want to make sure I'm doing the right thing. Thanks! — mclyne
I suggest that you open the csv file in notepad and look at your numbers columns. Do the columns contain numbers with commas? — chinsoon12

Mighty God Loki Mighty God Loki · Accepted Answer · 2018-02-17T09:18:47

Please check if you replaced your NAs with 0 or '0' as one is character and other is int. Or you can even try using as.numeric(column_name) function to convert your char 0s with int 0. Also this error occurs if your dataset has factors, because those are not int values corrplot throws this error. It would be helpful of you put sample of your data in the question using

str(head(your_dataset))

That would be helpful for you to check the datatypes of columns. Let me know if I am wrong. Cheerio.

R: errors in cor() and corrplot()

1 Answers