Python - Accelerometer reading and writing to CSV file at 1 kHz rate

Question

I am trying to use a MPU-6000 accelerometer and Raspberry Pi Zero W to log vibration data in a windshield. I'm fairly new to Python so please bear with me.

I've written a python2 script that configures the MPU-6000 to communicate over I2C, with the clock configured to 400 kHz. The MPU-6000 gives an interrupt when there is new data available in the accelerometer registers, which is read, converted to 2's complement and then written to a CSV file together with a timestamp. The output rate of the accelerometer is configured to be 1 kHz.

I'm experiencing that when sampling all three sensor axis the script isn't able to write all data points to the CSV file. Instead of a 1000 datapoints pr axis pr second I get approximately 650 datapoints pr axis pr second. I've tried writing only one axis, which proved succesfull with 1000 datapoints pr second. I know that the MPU-6000 has a FIFO register available, which I probably can burst read to get 1000 samples/s without any problem. The problem will be obtaining a timestamp for each sample, so I haven't tried to implement reading from the FIFO register yet.

I will most likely do most of the post processing in Matlab, so the most important things the python script should do is to write sensor data in any form to a CSV file at the determined rate, with a timestamp.

Is there any way to further improve my Python script, so I can sample all three axis and write to a CSV file at a 1 kHz rate?

Parts of my script is depicted below:

#!/usr/bin/python
import smbus
import math
import csv
import time
import sys
import datetime

# Register addresses
power_mgmt_1 = 0x6b
power_mgmt_2 = 0x6c
samlerate_divider = 0x19
accel_config = 0x1C
INT_Enable = 0x38



def read_byte(reg):
    return bus.read_byte_data(address, reg)

def read_word(reg):
    h = bus.read_byte_data(address, reg)
    l = bus.read_byte_data(address, reg+1)
    value = (h <<8)+l
    return value

def read_word_2c(reg):
    val = read_word(reg)

    if (val >= 0x8000):
        return -((65535 - val) + 1)
    else:
        return val



csvwriter = None

def csv_open():
    csvfile = open('accel-data.csv', 'a')

    csvwriter = csv.writer(csvfile)




def csv_write(timedelta, accelerometerx, accelerometery, accelerometerz):

    global csvwriter

    csvwriter.writerow([timedelta,  accelerometerx, accelerometery, 
    accelerometerz])



# I2C configs
bus = smbus.SMBus(1) 
address = 0x69       

#Power management configurations
bus.write_byte_data(address, power_mgmt_1, 0)
bus.write_byte_data(address, power_mgmt_2, 0x00)


#Configure sample-rate divider
bus.write_byte_data(address, 0x19, 0x07)


#Configure data ready interrupt:
bus.write_byte_data(address,INT_Enable, 0x01) 



#Opening csv file and getting ready for writing
csv_open()

csv_write('Time', 'X_Axis', 'Y_Axis', 'Z_Axis') 

print
print "Accelerometer"
print "---------------------"

print "Printing acccelerometer data: "

#starttime = datetime.datetime.now()

while True:


    data_interrupt_read =  bus.read_byte_data(address, 0x3A)

    if data_interrupt_read == 1:


        meas_time = datetime.datetime.now()
#       delta_time = meas_time - starttime


        accelerometer_xout = read_word_2c(0x3b)
        accelerometer_yout = read_word_2c(0x3d)
        accelerometer_zout = read_word_2c(0x3f)

#       accelerometer_xout = read_word(0x3b)
#       accelerometer_yout = read_word(0x3d)
#       accelerometer_zout = read_word(0x3f)

#       accelerometer_xout_scaled = accelerometer_xout / 16384.0
#       accelerometer_yout_scaled = accelerometer_yout / 16384.0
#       accelerometer_zout_scaled = accelerometer_zout / 16384.0



#       csv_write(meas_time, accelerometer_xout_scaled, 
        accelerometer_yout_scaled, accelerometer_zout_scaled)

        csv_write(meas_time, accelerometer_xout, accelerometer_yout, 
        accelerometer_zout) 


    continue

Please edit your question to fix the indentation of your code because it is (as I am sure you know) part of the Python language and its absence affects the meaning of your code! — balmy
I would recommend writing the data in your own binary format (not CSV), this would be far faster as there would be less processing and less bytes written to a file. You could use a separate script afterwards to convert the binary format into CSV if needed. Take a look at the struct library. — Martin Evans
A good test might be to comment out the write to CSV and see if your code reads all three channels at the rate you are aiming for. You are writing synchronously with reading - you might be able to use python threads or multiprocessing to separate the two activities - you could “write” csv to a string and only write to disk every second, or you could use @MartinEvans suggestion to write binary data and convert to csv later. — balmy
Yes, I am aware of the indent of code in Python, corrected now. The indent is correct in my text editor, but was missed when I exported the code to StackOverflow. Binary format could be a solution, maybe I will give it a test today. As of now my method to determine the sample rate is to see the time stamp according to the line number in the csv file that is generated. I'll probably implement a pin toggle just to determine if all three channels are read at my desired rate. Threading and multiprocessing are new subjects for me as well, I'll have to check its documentation further out. — Petter
Another way to allow for variable data sampling would be to record a high-resolution timestamp with the readings - then you don’t have to guess or assume 1ms/sample. In fact you could record timestamps before/after to get better precision for when the samples were actually taken. — balmy

Martin Evans Martin Evans · Accepted Answer · 2018-11-21T15:21:11

If the data you are trying to write is continuous, then the best approach is to minimise the amount of processing needed to write it and to also minimise the amount of data being written. To do this, a good approach would be to write the raw data into a binary formatted file. Each data word would then only require 2 bytes to be written. The datetime object can be converted into a timestamp which would need 4 bytes. So you would use a format such as:

[4 byte timestamp][2 byte x][2 byte y][2 byte z]

Python's struct library can be used to convert multiple variables into a single binary string which can be written to a file. The data appears to be signed, if this is the case, you could try writing the word as is, and then using the libraries built in support for signed values to read it back in later.

For example, the following could be used to write the raw data to a binary file:

#!/usr/bin/python
import smbus
import math
import csv
import time
import sys
import datetime
import struct

# Register addresses
power_mgmt_1 = 0x6b
power_mgmt_2 = 0x6c
samlerate_divider = 0x19
accel_config = 0x1C
INT_Enable = 0x38


def read_byte(reg):
    return bus.read_byte_data(address, reg)


def read_word(reg):
    h = bus.read_byte_data(address, reg)
    l = bus.read_byte_data(address, reg+1)
    value = (h <<8)+l
    return value


# I2C configs
bus = smbus.SMBus(1) 
address = 0x69       

#Power management configurations
bus.write_byte_data(address, power_mgmt_1, 0)
bus.write_byte_data(address, power_mgmt_2, 0x00)

#Configure sample-rate divider
bus.write_byte_data(address, 0x19, 0x07)

#Configure data ready interrupt:
bus.write_byte_data(address, INT_Enable, 0x01) 


print
print "Accelerometer"
print "---------------------"

print "Printing accelerometer data: "

#starttime = datetime.datetime.now()
bin_format = 'L3H'

with open('accel-data.bin', 'ab') as f_output:
    while True:
        #data_interrupt_read =  bus.read_byte_data(address, 0x3A)
        data_interrupt_read = 1

        if data_interrupt_read == 1:
            meas_time = datetime.datetime.now()
            timestamp = time.mktime(meas_time.timetuple())

            accelerometer_xout = read_word(0x3b)
            accelerometer_yout = read_word(0x3d)
            accelerometer_zout = read_word(0x3f)

            f_output.write(struct.pack(bin_format, timestamp, accelerometer_xout, accelerometer_yout, accelerometer_zout))

Then later on, you could then convert the binary file to a CSV file using:

from datetime import datetime
import csv
import struct

bin_format = 'L3h'  # Read data as signed words
entry_size = struct.calcsize(bin_format)

with open('accel-data.bin', 'rb') as f_input, open('accel-data.csv', 'wb') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(['Time', 'X_Axis', 'Y_Axis', 'Z_Axis'])

    while True:
        bin_entry = f_input.read(entry_size)

        if len(bin_entry) < entry_size:
            break

        entry = list(struct.unpack(bin_format, bin_entry))
        entry[0] = datetime.fromtimestamp(entry[0]).strftime('%Y-%m-%d  %H:%M:%S')
        csv_output.writerow(entry)

If your data collection is not continuous, you could make use of threads. One thread would read your data into a special queue. Another thread could read items out of the queue onto the disk.

If it is continuous this approach will fail if the writing of data is slower than the reading of it.

Take a look at the special Format characters used to tell struct how to pack and unpack the binary data.

Python - Accelerometer reading and writing to CSV file at 1 kHz rate

1 Answers