Create a matrix of possible combinations with itertools

Question

I have a list which length is 1442. Every element of the list is a list each contains 10 data points. I have to calculate the correlation of every possible 2 long combinations, and then find groups that correlate the most.

Data is available here:

import pandas as pd
import numpy as np
import datetime
import math
import itertools
import matplotlib.pyplot as plt
pd.set_option('display.max_rows', 50)
pd.set_option('display.max_columns', 50)

df_15m = pd.read_csv(r'.../USDT_BTC 15-Minute.csv')
df_15m.head()

df_15m['date'] = df_15m['date'].apply(lambda x: 
datetime.datetime.fromtimestamp(x).strftime('%Y-%m-%d %H:%M:%S'))
df_15m['day'] = df_15m['date'].apply(lambda x: 
datetime.datetime.strptime(x,'%Y-%m-%d %H:%M:%S').strftime('%Y-%m-%d'))

df_15m.set_index('date',inplace=True)
df_15m.index = pd.to_datetime(df_15m.index)
del df_15m.index.name


t = [0]

for i in range(len(df_15m)-1):

    p1 = df_15m.iloc[i+1]['weightedAverage']
    p0 = df_15m.iloc[i]['weightedAverage']
    t.append(math.log(p1/p0))

df_15m['BVOL15M_INDEX'] = t

by_day_vol = 
pd.DataFrame(df_15m['BVOL15M_INDEX'].resample('H').std()*math.sqrt(24))
by_day_price = pd.DataFrame(df_15m['weightedAverage'].resample('H').mean())
res = pd.merge(by_day_price, by_day_vol, left_index=True, right_index=True)

#creating subsets with 10 data points
df_ = res[:-6]
n = 10
list_df = [df_['BVOL15M_INDEX'][i:i+n] for i in range(0, df_.shape[0], n)]


l = []

for subset in itertools.combinations(list_df, 2):

    corrcoef = np.corrcoef(subset[0], subset[1])[1,0]
    l.append(corrcoef)

l contains the correlations for all possible combinations. I'd like to create a matrix, which ixj position contains the correlation for corresponding groups from list_df. For the final result, i should be able to group periods which are correlating.

Can someone please help me, or provide me an easier solution?

Thank you!

zabop zabop · Accepted Answer · 2018-09-05T11:13:38

What about something like this?

#lets have 3 lists:
list=[None]*3
list[0]=[1, 2, 3]
list[1]=[3, 4, 5]
list[2]=[5, 6, 7]

yourmat=np.matrix([list[x] for x in range(3)])

plt.imshow(yourmat, cmap='hot', interpolation='nearest')
plt.show()

This produces an output:

Of course, replace my lists with yours. The code above can be scaled up to 1442 lists (change the 3 in the range(3) to as many lists as you are using).

Create a matrix of possible combinations with itertools

2 Answers