How can I reduce delay when sending audio over UDP?

Question

I've written a small java program (pc) and android app which is meant to stream audio from the pc to my phone. I am sending the audio data via UDP, and am having trouble determining the source of the delay which occurs. From client to server, about 1.5-2 seconds pass before the packets are processed.

My program buffers 512 bytes to send to the client, which are played back immediately upon reception.

I tested some different configurations, with anything (substantially) higher than this value, delays only increase. With lower values, there is no noticeable improvement in terms of latency, but there is noticeable quality loss.

The ping time between the devices is only 3ms according to windows, so I am assuming that the network connection is not the issue, though I am not positive.

My Client (PC)'s code is shown below.

byte[] buffer = new byte[512];
while (true) {
    try {
        audioInput.read(buffer, 0, buffer.length);
        DatagramPacket data = new DatagramPacket(buffer, buffer.length, address, port);
        dout.send(data);
        System.out.println("Sent Packet #" + i++);
    } catch (IOException e) {
        e.printStackTrace();
    }
}

The Server (Phone)'s code is as follows.

    byte[] buffer = new byte[512];
    DatagramPacket incomingPacket = new DatagramPacket(buffer, buffer.length);
    while (true) {
        try {
            dgs.receive(incomingPacket);
            buffer = incomingPacket.getData();
            audioOutput.write(buffer, 0, buffer.length);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

I expected packets to arrive at a speed closer to the network delay of <5ms, but actually receive them only after ~1500ms.

I was hoping some of you may have experience with this sort of issue. I know that apps like Discord and Skype stream at higher bitrates, with much higher latency, but have a substantially lower delay, so I was hoping there might have been something that I missed.

is byte[] buffer = new byte[512]; the 512 byte buffer you're talking about in the question? — Ryotsu
Looks more like a network problem to me. Use a network test tool like iperf to test your network latency between PC and phone. Also check the Wifi for frequency collisions with neighboring Wifi networks. — Robert

marko marko · Accepted Answer · 2019-01-05T12:01:44

You are attempting to do real-time streaming. Let's look at the latency budget all the way through the process.

Firstly, you are using 512 byte buffers - assuming the samples are floats and a sample rate of 44.1k, you process 128 samples each time, this gives a buffer period of ~2.9ms.

The best you can get is ~7.5ms at this buffer size.

The flow of control:

1: Audio input hardware generates samples and stores them in a buffer. After a pre-determined number of samples, the operating system is interrupted to service audio. This buffer period may be significantly longer than 128 samples. This is T1

2: The operating system schedules the audio daemon process to run. It waits S1 before running

3: The audio daemon runs, processes the audio, writes it to a buffer, and signals to the operating system that samples are available to be read. This time is negligibly short.

4: The operating system schedules your input process to run by unblocking the call to audioInput.read(buffer, 0, buffer.length);. It waits S2 before getting scheduled.

5: You call System.out.println(). This potentially blocks - especially as you're writing to it ~350 times a seconds - probably until another unrelated process gets scheduled. S3

6: A UDP is written to a network socket. The OS may queue it for transmit - which is likely to happen a negligibly short time later

7: Transit across the network. T2

The receiving side is pretty much the reverse of the above.

Thus total latency is

2 * (T1 + S1 + S2 + S3 + T2)

I imagine that most of your delay is the hardware buffering period at both send and receive - T1. If you get the much lower than 10ms, S1,S2,S3 are going to start to become significant.

Note:

S1, S2 are operating scheduling delays and dependant on system load and scheduler policy. Audio render handlers are typically run with real-time threa priority.
You can eliminate S3 by not logging to the console. This latency is particularly unpredictable.
The Java runtime may impose some hidden runtime costs (eg. GC). This will be a limiting factor to reliable low-latency audio.

The recipe for really low latency is:

Implement in C or C++
No memory allocation, logging in render loop
Pin stack and heap pages to prevent them getting swapped
Run audio render threads with real-time scheduling priority.
Lockless data structures to prevent priority inversion.

How can I reduce delay when sending audio over UDP?

1 Answers