Realtime streaming with QAudioOutput

Question

I am working on a C++ project to read/process/play raw audio from a microphone array system, with its own C++ API. I am using Qt to program the software.

From this post about Real Time Streaming With QAudioOutput (Qt), I wanted to follow up and ask for advice about what to do if the Raw Audio Data comes from a function call that takes about 1000ms (1 sec) to process? How would I still be able to achieve the real time audio playback.

It takes about about a second to process because I had read that when writing to QIODevice::QAudioFormat->start(); it is advisable to use a period's worth of bytes to prevent buffer underrun / overrun. http://cell0907.blogspot.sg/2012/10/qt-audio-output.html

I have set up a QByteArray and QDataStream to stream the data received from the function call.

The API is CcmXXX()
Reading the data from the microphone array returns an array of 32 bit integers
Of the 32 bit integers, 24 bits resolution, 8 bits LSB padded zeros.
It comes in block sizes (set at 1024 samples) x 40 microphones
Each chunk writes about one block, till the number of bytes written reaches close to the period size / free amount of bytes.

Tested: Connected my slots to a notify of about 50ms, to write one period worth of bytes. QByteArray in circular buffer style. Added a mutex lock/unlock at the read/write portions.

Result: Very short split ms of actual audio played, lots of jittering and non-recorded sounds.

Please do offer feedback on how I could improve my code.

Setting up QAudioFormat

void MainWindow::init_audio_format(){
            m_format.setSampleRate(48000); //(8000, 11025, 16000, 22050, 32000, 44100, 48000, 88200, 96000, 192000
            m_format.setByteOrder(QAudioFormat::LittleEndian);
            m_format.setChannelCount(1);
            m_format.setCodec("audio/pcm");
            m_format.setSampleSize(32); //(8, 16, 24, 32, 48, 64)
            m_format.setSampleType(QAudioFormat::SignedInt); //(SignedInt, UnSignedInt, Float)

            m_device = QAudioDeviceInfo::defaultOutputDevice();
            QAudioDeviceInfo info(m_device);
            if (!info.isFormatSupported(m_format)) {
                qWarning() << "Raw audio format not supported by backend, cannot play audio.";
                return;
            }
}

Initialising Audio and QByteArray/Datastream

void MainWindow::init_audio_output(){   
    m_bytearray.resize(65536);
    mstream = new QDataStream(&m_bytearray,QIODevice::ReadWrite);
    mstream->setByteOrder(QDataStream::LittleEndian);
    audio = new QAudioOutput(m_device,m_format,this);
    audio->setBufferSize(131072);
    audio->setNotifyInterval(50);
    m_audiodevice = audio->start();
    connect(audio,SIGNAL(notify()),this,SLOT(slot_writedata()));
    read_frames();
}

Slot:

void MainWindow::slot_writedata(){
    QMutex mutex;
    mutex.lock();
    read_frames();
    mutex.unlock();
}

To read the frames:

    void MainWindow::read_frames(){
        qint32* buffer;
        int frameSize, byteCount=0;
        DWORD tdFrames, fdFrames;
        float fvalue = 0;
        qint32 q32value;

        frameSize = 40 * mBlockSize; //40 mics 
        buffer = new int[frameSize];
        int periodBytes = audio->periodSize();
        int freeBytes = audio->bytesFree();
        int chunks = qMin(periodBytes/mBlockSize,freeBytes/mBlockSize);

        CcmStartInput();

        while(chunks){
            CcmReadFrames(buffer,NULL,frameSize,0,&tdFrames,&fdFrames,NULL,CCM_WAIT);
            if(tdFrames==0){
                break;
            }
            int diffBytes = periodBytes - byteCount;

            if(diffBytes>=(int)sizeof(q32value)*mBlockSize){
                for(int x=0;x<mBlockSize;x++){
                    q32value = (quint32)buffer[x]/256;
                   *mstream << (qint32)fvalue;
                    byteCount+=sizeof(q32value);
                }
            }
            else{
                for(int x=0;x<(diffBytes/(int)sizeof(q32value));x++){
                    q32value = (quint32)buffer[x]/256;
                    *mstream << (qint32) fvalue;
                    byteCount+=sizeof(q32value);
                }
            }
            --chunks;
        }
        CcmStopInput();
        mPosEnd = mPos + byteCount;
        write_frames();
        mPos += byteCount;
        if(mPos >= m_bytearray.length()){
              mPos = 0;
              mstream->device()->seek(0); //change mstream pointer back to bytearray start
        }
    }

To write the frames:

void MainWindow::write_frames()
{
    int len = m_bytearray.length() - mPos;
    int bytesWritten = mPosEnd - mPos;

    if(len>=audio->periodSize()){
        m_audiodevice->write(m_bytearray.data()+mPos, bytesWritten);
    }
    else{

        w_data.replace(0,qAbs(len),m_bytearray.data()+mPos);
        w_data.replace(qAbs(len),audio->periodSize()-abs(len),m_bytearray.data());
       m_audiodevice->write(w_data.data(),audio->periodSize());
    }
}

dtech dtech · Accepted Answer · 2017-12-27T06:39:50

Audio support in Qt is actually quite rudimentary. The goal is to have media playback at the lowest possible implementation and maintenance cost. The situation is especially bad on windows, where I think the ancient MME API is still employed for audio playback.

As a result, the Qt audio API is very far from realtime, making it particularly ill-suited for such applications. I recommend using portaudio or rtaudio, which you can still wrap in Qt style IO devices if you will. This will give you access to better performing platform audio APIs and much better playback performance at very low latency.

Realtime streaming with QAudioOutput

1 Answers