Loading and writing long text data in Python

0
votes

I have a long electrocardiogram (ECG) data written in .txt format. The .txt file size is usually 2 GB large because it is a recording of more than 7 days sampled at 250 Hz.

The data format of the .txt file is:

  • column 1: the timestamp of each sample;
  • column 2: the ECG value in integer for the uV.

I would like to know:

  1. How to load (efficiently) a portion of a large and long text file in Python without blowing the RAM? For example, how can I jump to the 72nd hour (row 64,800,000) of the ECG recording and only load 10-min samples (150k samples) from it? Is there any Python module that can jump to specific rows of a long file?
  2. After I performed signal processing on some portion of the ECG data, how can I save it to another large and long text file? For example, after I completed some signal processing of the 10-min samples in question 1, should I concatenate the 150k length array to another text file? What if I want to insert this 150k length array in the middle of another long text file?

The use-case for me is that I would like to run some signal processing algorithms on a high-performance computer (HPC) with a fast read-write speed. I only have experience with MATLAB. I could not figure out an efficient way of processing the long text file in MATLAB, so I had to cut the 7-day recording into an hour-long file (900k length array) and load one file at a time if I needed to process ECG within that specific hour.

Thank you for your help in advance!

pythonlarge-files
You can't jump DIRECTLY to a text line without having read all of the lines first. However, if the lines are all roughly the same size, you can do a binary search. You can use file.seek() to jump to 64800000*(avg length) and then refine the guess. However, it SOUNDS like what you really need is a database. You can't write into the middle of a text file. - Tim Roberts