How can i sort Binance historical candles for multiple pairs across multiple timeframes

Question

I'm downloading historical candlestick data for multiple crypto pairs across different timeframes from the binance api, i would like to know how to sort this data according to pair and timeframe and check which pair on which timeframe executes my code, the following code is what i use to get historical data

import requests

class BinanceFuturesClient:

def __init__(self):
    self.base_url = "https://fapi.binance.com"

def make_requests(self, method, endpoint, data):
    if method=="GET":
        response = requests.get(self.base_url + endpoint, params=data)
    return response.json()

def get_symbols(self):
    symbols = []
    exchange_info = self.make_requests("GET", "/fapi/v1/exchangeInfo", None)
    if exchange_info is not None:
        for symbol in exchange_info['symbols']:
            if symbol['contractType'] == 'PERPETUAL' and symbol['quoteAsset'] == 'USDT':
                symbols.append(symbol['pair'])

    return symbols

def initial_historical_data(self, symbol, interval):
    data = dict()
    data['symbol'] = symbol
    data['interval'] = interval
    data['limit'] = 35
    raw_candle = self.make_requests("GET", "/fapi/v1/klines", data)
    candles = []
    if raw_candle is not None:
        for c in raw_candle:
            candles.append(float(c[4]))
    return candles[:-1]

running this code

print(binance.initial_historical_data("BTCUSDT", "5m"))

will return this as the output

[55673.63, 55568.0, 55567.89, 55646.19, 55555.0, 55514.53, 55572.46, 55663.91, 55792.83, 55649.43, 
55749.98, 55680.0, 55540.25, 55470.44, 55422.01, 55350.0, 55486.56, 55452.45, 55507.03, 55390.23, 
55401.39, 55478.63, 55466.48, 55584.2, 55690.03, 55760.81, 55515.57, 55698.35, 55709.78, 55760.42, 
55719.71, 55887.0, 55950.0, 55980.47]

which is a list of closes

i want to loop through the code in such a manner that i can return all the close prices for the pairs and timeframes i need and sort it accordingly, i did give it a try but am just stuck at this point

period = ["1m", "3m", "5m", "15m"]
binance = BinanceFuturesClient()
symbols = binance.get_symbols()
for symbol in symbols:
    for tf in period:
        historical_candles = binance.initial_historical_data(symbol, tf)
        # store values and run through strategy

mde mde · Accepted Answer · 2021-05-28T11:43:22

You can use my code posted below. It requires python-binance package to be installed on your environment and API key/secret from your Binance account. Method tries to load data by weekly chunks (parameter step) and supports resending requests on failures after timeout. It may helps when you need to fetch huge amount of data.

import pandas as pd
import pytz, time, datetime
from binance.client import Client
from tqdm.notebook import tqdm

def binance_client(api_key, secret_key):
    return Client(api_key=api_key, api_secret=secret_key)

def load_binance_data(client, symbol, start='1 Jan 2017 00:00:00', timeframe='1M', step='4W', timeout_sec=5):
    tD = pd.Timedelta(timeframe)
    now = (pd.Timestamp(datetime.datetime.now(pytz.UTC).replace(second=0)) - tD).strftime('%d %b %Y %H:%M:%S')
    tlr = pd.DatetimeIndex([start]).append(pd.date_range(start, now, freq=step).append(pd.DatetimeIndex([now])))

    print(f' >> Loading {symbol} {timeframe} for [{start}  -> {now}]')
    df = pd.DataFrame()
    s = tlr[0]
    for e in tqdm(tlr[1:]):
        if s + tD < e:
            _start, _stop = (s + tD).strftime('%d %b %Y %H:%M:%S'), e.strftime('%d %b %Y %H:%M:%S')
            nerr = 0
            while nerr < 3:
                try:
                    chunk = client.get_historical_klines(symbol, timeframe.lower(), _start, _stop)
                    nerr = 100
                except e as Exception:
                    nerr +=1
                    print(red(str(e)))
                    time.sleep(10)

            if chunk:
                data = pd.DataFrame(chunk, columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'close_time', 'quote_av', 'trades', 'tb_base_av', 'tb_quote_av', 'ignore' ])
                data.index = pd.to_datetime(data['timestamp'].rename('time'), unit='ms')
                data = data.drop(columns=['timestamp', 'close_time']).astype(float).astype({
                    'ignore': bool,
                    'trades': int,
                })
                df = df.append(data)
            s = e
            time.sleep(timeout_sec)

    return df

How to use

c = binance_client(<your API code>, <your API secret>)
# loading daily data from 1/Mar/21 till now (your can use other timerames like 1m, 5m etc)
data = load_binance_data(c, 'BTCUSDT', '2021-03-01', '1D')

It returns indexed DataFrame with loaded data:

time	open	high	low	close	volume	quote_av	trades	tb_base_av	tb_quote_av	ignore
2021-03-02 00:00:00	49595.8	50200	47047.6	48440.7	64221.1	3.12047e+09	1855583	31377	1.52515e+09	False
2021-03-03 00:00:00	48436.6	52640	48100.7	50349.4	81035.9	4.10952e+09	2242131	40955.4	2.07759e+09	False
2021-03-04 00:00:00	50349.4	51773.9	47500	48374.1	82649.7	4.07984e+09	2291936	40270	1.98796e+09	False
2021-03-05 00:00:00	48374.1	49448.9	46300	48751.7	78192.5	3.72713e+09	2054216	38318.3	1.82703e+09	False
2021-03-06 00:00:00	48746.8	49200	47070	48882.2	44399.2	2.14391e+09	1476474	21500.6	1.03837e+09	False

Next steps are up to you and dependent on how would you like to design your data structure. In simplest case you could store data into dictionaries:

from collections import defaultdict
data = defaultdict(dict)

for symbol in ['BTCUSDT', 'ETHUSDT']:
    for tf in ['1d', '1w']:
        historical_candles = load_binance_data(c, symbol, '2021-05-01', timeframe=tf)

        # store values and run through strategy
        data[symbol][tf] = historical_candles

to get access to your OHLC you just need following: data['BTCUSDT']['1d'] etc.

How can i sort Binance historical candles for multiple pairs across multiple timeframes

1 Answers

How to use