How to optimize ASCII output with QTextStream

Question

I'm currently writing out billions of binary records to ASCII files (ugh). I've got things working just fine, but I'd like to optimize the performance if I can. The problem is, the user is allowed to select any number of fields to output, so I can't know at compile-time which of 3-12 fields they'll include.

Is there a faster way to construct lines of ASCII text? As you can see, the types of the fields vary quite a bit and I can't think of a way around the series of if() statements. The output ASCII file has one line per record, so I've tried using a template QString constructed with arg, but that just slowed things down about 15%.

A faster solution doesn't have to use QTextStream, or necessarily write directly to the file, but the output is too large to write the whole thing to memory.

Here's some sample code:

QFile outfile(outpath);
if(!outfile.open(QIODevice::WriteOnly | QIODevice::Text | QIODevice::Truncate))
{
    qWarning("Could not open ASCII for writing!");
    return false;
} else
{
    /* compute XYZ precision */
    int prec[3] = {0, 0, 0}; //these non-zero values are determined programmatically

    /* set up the writer */
    QTextStream out(&outfile);
    out.setRealNumberNotation(QTextStream::FixedNotation);
    out.setRealNumberPrecision(3);
    QString del(config.delimiter); //the user chooses the delimiter character (comma, tab, etc) - using QChar is slower since it has to be promoted to QString anyway

    /* write the header line */
    out << "X" << del << "Y" << del << "Z";
    if(config.fields & INTFIELD)
        out << del << "IntegerField";
    if(config.fields & DBLFIELD)
        out << del << "DoubleField";
    if(config.fields & INTFIELD2)
        out << del << "IntegerField2";
    if(config.fields & TRIPLEFIELD)
        out << del << "Tri1" << del << "Tri2" << del << "Tri3";
    out << "\n";

    /* write out the points */
    for(quint64 ptnum = 0; ptnum < numpoints; ++ptnum)
    {
        pt = points.at(ptnum);
        out.setRealNumberPrecision(prec[0]);
        out << pt->getXYZ(0);
        out.setRealNumberPrecision(prec[1]);
        out << del << pt->getXYZ(1);
        out.setRealNumberPrecision(prec[2]);
        out << del << pt->getXYZ(2);
        out.setRealNumberPrecision(3);
        if(config.fields & INTFIELD)
            out << del << pt->getIntValue();
        if(config.fields & DBLFIELD)
            out << del << pt->getDoubleValue();
        if(config.fields & INTFIELD2)
            out << del << pt->getIntValue2();
        if(config.fields & TRIPLEFIELD)
        {
            out << del << pt->getTriple(0);
            out << del << pt->getTriple(1);
            out << del << pt->getTriple(2);
        }
        out << "\n";
    } //end for every point
outfile.close();

You need to profile your app to find out what slows down it. Qt classes are already optimized and work fast if properly used. And your code is correct, I see nothing obviously slow. Profiling is really required in your case. Maybe your disk is slow, maybe it's QTextStream, maybe it's QString. — Pavel Strakhov
I have yet to find a good Qt profiler for 64bit windows. VerySleepy has potential, but its output is so arcane I hardly understand it. Suggestions? — Phlucious
It depends on which compiler you're using. I like gprof for gcc. For MSVC compiler the standard vsprofiler can be used. — Pavel Strakhov

Nikos C. Nikos C. · Accepted Answer · 2013-07-23T16:31:14

(This doesn't answer the profiler question. It tries to answer the original question, which is the performance issue.)

I would suggest avoiding the use of QTextStream altogether in this case to see if that helps. The reason it might help with performance is that there's overhead involved, because text is encoded internally to UTF-16 for storage, and then decoded again to ASCII or UTF-8 when writing it out. You have two conversions there that you don't need.

Try using only the standard C++ std::ostringstream class instead. It's very similar to QTextStream and only minor changes are needed in your code. For example:

#include <sstream>

// ...

QFile outfile(outpath);
if (!outfile.open(QIODevice::WriteOnly | QIODevice::Text
                | QIODevice::Truncate))
{
    qWarning("Could not open ASCII for writing!");
    return false;
}

/* compute XYZ precision */
int prec[3] = {0, 0, 0};

std::ostringstream out;
out.precision(3);
std::fixed(out);
// I assume config.delimiter is a QChar.
char del = config.delimiter.toLatin1();

/* write the header line */
out << "X" << del << "Y" << del << "Z";
if(config.fields & INTFIELD)
    out << del << "IntegerField";
if(config.fields & DBLFIELD)
    out << del << "DoubleField";
if(config.fields & INTFIELD2)
    out << del << "IntegerField2";

if(config.fields & TRIPLEFIELD)
    out << del << "Tri1" << del << "Tri2" << del << "Tri3";
out << "\n";

/* write out the points */
for(quint64 ptnum = 0; ptnum < numpoints; ++ptnum)
{
    pt = points.at(ptnum);
    out.precision(prec[0]);
    out << pt->getXYZ(0);
    out.precision(prec[1]);
    out << del << pt->getXYZ(1);
    out.precision(prec[2]);
    out << del << pt->getXYZ(2);
    out.precision(3);
    if(config.fields & INTFIELD)
        out << del << pt->getIntValue();
    if(config.fields & DBLFIELD)
        out << del << pt->getDoubleValue();
    if(config.fields & INTFIELD2)
        out << del << pt->getIntValue2();
    if(config.fields & TRIPLEFIELD)
    {
        out << del << pt->getTriple(0);
        out << del << pt->getTriple(1);
        out << del << pt->getTriple(2);
    }
    out << "\n";

    // Write out the data and empty the stream.
    outfile.write(out.str().data(), out.str().length());
    out.str("");
}
outfile.close();

How to optimize ASCII output with QTextStream

6 Answers