I want to know more about this as this is new for me..
I am trying to query InfluxDB with python to fetch data in 5 min time interval. I used a simple for-loop to get my data in small chunks and appended the chunks into another empty dataframe inside for loop one after another. This worked out pretty smoothly and I see my output. But while I try to perform mathematical operations on this large dataframe , it gives me a Memory error stated below:
"Memory Error : Unable to allocate 6.95GiB for an array with shape (993407736) and datatype int64"
My system has these info 8.00GB RAM, 64 bit OS x64 based processor. Could my system be not supporting this ?
Is there an alternate way I can append small dataframes into another dataframe without these memory issues. I am new to this data stuff with python and I need to work with this large chunk of data.... may be an year