linear regression - how to check for correlation among continuous and categorical variables in python?

Question

I have a dataset including categorical variables(binary) and continuous variables. I'm trying to apply a linear regression model for predicting a continuous variable. Can someone please let me know how to check for correlation among the categorical variables and the continuous target variable.

Current Code:

import pandas as pd
df_hosp = pd.read_csv('C:\Users\LAPPY-2\Desktop\LengthOfStay.csv')

data = df_hosp[['lengthofstay', 'male', 'female', 'dialysisrenalendstage', 'asthma', \
              'irondef', 'pneum', 'substancedependence', \
              'psychologicaldisordermajor', 'depress', 'psychother', \
              'fibrosisandother', 'malnutrition', 'hemo']]
print data.corr()

All of the variables apart from lengthofstay are categorical. Should this work?

What have your tried so far? Provide us with the code and clearly mention where you're having the issue. — Adeel Ahmad
Look for ANOVA in python (in R would "aov"). This helps you identify, if the means (continous values) of the different groups (categorical values) have signficant differnt means. If you have only two groups, use a two-sided t.test (paired or unpaired). — Rockbar
Follow this tutorial. I think that is what you are looking for: marsja.se/four-ways-to-conduct-one-way-anovas-using-python — Rockbar
@AdeelAhmad I've added the code that i've got so far. the output that i got was a matrix but I'm not sure if that is correct or not. For continuous variables this works well, as far as I know. — funnyguy
Thanks @Rockbar, but I have the data in a pandas dataframe and there are multiple columns with huge number of observations. Would Anova be good here? — funnyguy

glegoux glegoux · Accepted Answer · 2017-06-22T09:01:29

Convert your categorical variable into dummy variables here and put your variable in numpy.array. For example:

data.csv:

age,size,color_head
4,50,black
9,100,blonde
12,120,brown
17,160,black
18,180,brown

Extract data:

import numpy as np
import pandas as pd

df = pd.read_csv('data.csv')

df:

Convert categorical variable color_head into dummy variables:

df_dummies = pd.get_dummies(df['color_head'])
del df_dummies[df_dummies.columns[-1]]
df_new = pd.concat([df, df_dummies], axis=1)
del df_new['color_head']

df_new:

Put that in numpy array:

x = df_new.values

Compute the correlation:

correlation_matrix = np.corrcoef(x.T)
print(correlation_matrix)

Output:

array([[ 1.        ,  0.99574691, -0.23658011, -0.28975028],
       [ 0.99574691,  1.        , -0.30318496, -0.24026862],
       [-0.23658011, -0.30318496,  1.        , -0.40824829],
       [-0.28975028, -0.24026862, -0.40824829,  1.        ]])

See :

numpy.corrcoef

linear regression - how to check for correlation among continuous and categorical variables in python?

2 Answers