Real-time Audio processing - latency feasibility check

3
votes

I have an application concept that required real-time audio signal processing that can be broadly described as: a) sampling incoming audio (from microphone), b) performing signal processing functions (such as filtering, fourier transform, filtering and manipulation, inverse fourier transform) c) play-out (via speaker jack)

I believe that the "end to end" round trip timing (a) to (c) would need be in the order of 2 to 5 ms for the application to work in the real-world.

So, my question is this possible on today's generation of iphones and android phones?

1
audiosignalsreal-timelatencyfeasibility
5 ms at, say, 44.1 kHz restricts you to a maximum FFT length of 220 samples. – Oliver Charlesworth
I believe just the I/O is on that order for iOS (very roughly speaking). Android used to be much higher, but newer versions are supposed to be better. – Bjorn Roche
@Oli Charlesworth - That is true if only operation in the users 'end to end' audio processing pipeline is FFT. But there is going to be A/D, then there is some other signal processing as mentioned by OP, and then finally there is D/A. So practical FFT length could be less than 220 samples for 5ms latency budget mentioned in OP. – goldenmean
@goldenmean: Of course. But my point was that this is an absolute upper limit on the FFT size, to illustrate just how short 5ms is... – Oliver Charlesworth

1 Answers

1
votes

On iOS, it is possible, but not guaranteed. I have managed to get ~6ms (22050 sampling rate, 128 samples buffer size) in my iOS app which does real-time processing of speech input. Take a look at Novocaine (https://github.com/alexbw/novocaine) - which provides a nice encapsulation of Audio Units and makes programming easier.

However, keep in mind that even if you request a particular buffer size, at run time iOS may decide to send larger buffers at longer intervals (=> higher latency) based on resource constraints. For example, if you have requested a buffer size of 128 (~6ms), you MAY end up getting 256 size buffers at 12ms instead. Your app has to take this into account and handle the buffer accordingly.

On Android, unfortunately, low-latency round-trip audio is a much bigger problem. This is because latency is driven by a host of device/manufacturer driven factors like hardware/driver level buffers and these vary from device to device. You can find a discussion of this long-standing Android handicap here: https://code.google.com/p/android/issues/detail?id=3434

My suggestion would be to ignore Android for now, and implement/validate your signal processing algorithms on an iOS device. Later, you can consider porting them to Android.