QNetworkAccessManager: post http multipart from serial QIODevice

Question

I'm trying to use QNetworkAccessManager to upload http multiparts to a dedicated server.

The multipart consists of a JSON part describing the data being uploaded.

The data is read from a serial QIODevice, which encrypts the data.

This is the code that creates the multipart request:

QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);

QHttpPart metaPart;
metaPart.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
metaPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"metadata\""));
metaPart.setBody(meta.toJson());
multiPart->append(metaPart);

QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant(fileFormat));
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"file\""));
filePart.setBodyDevice(p_encDevice);
p_encDevice->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
multiPart->append(filePart);

QNetworkAccessManager netMgr;
QScopedPointer<QNetworkReply> reply( netMgr.post(request, multiPart) );
multiPart->setParent(reply.data()); // delete the multiPart with the reply

If the p_encDevice is an instance of QFile, that file gets uploaded just fine.

If the specialised encrypting QIODevice is used (serial device) then all of the data is read from my custom device. however QNetworkAccessManager::post() doesn't complete (hangs).

I read in the documentation of QHttpPart that:

if device is sequential (e.g. sockets, but not files), QNetworkAccessManager::post() should be called after device has emitted finished().

Unfortunately I don't know how do that.

Please advise.

EDIT:

QIODevice doesn't have finished() slot at all. What's more, reading from my custom IODevice doesn't happen at all if QNetworkAccessManager::post() is not called and therefore the device wouldn't be able to emit such an event. (Catch 22?)

EDIT 2:

It seems that QNAM does not work with sequential devices at all. See discussion on qt-project.

EDIT 3:

I managed to "fool" QNAM to make it think that it is reading from non-sequential devices, but seek and reset functions prevent seeking. This will work until QNAM will actually try to seek.

bool AesDevice::isSequential() const
{
    return false;
}

bool AesDevice::reset()
{
    if (this->pos() != 0) {
        return false;
    }
    return QIODevice::reset();
}

bool AesDevice::seek(qint64 pos)
{
    if (this->pos() != pos) {
        return false;
    }
    return QIODevice::seek(pos);
}

Answer 1

You'll need to refactor your code quite a lot so that the variables you pass to post are available outside that function you've posted, then you'll need a new slot defined with the code for doing the post inside the implementation. Lastly you need to do connect(p_encDevice, SIGNAL(finished()), this, SLOT(yourSlot()) to glue it all together.

You're mostly there, you just need to refactor it out and add a new slot you can tie to the QIODevice::finished() signal.

Answer 2

I've had more success creating the http post data manually than with using QHttpPart and QHttpMultiPart. I know it's probably not what you want to hear, and it's a little messy, but it definitely works. In this example I am reading from a QFile, but you can call readAll() on any QIODevice. It also is worth noting, QIODevice::size() will help you check if all the data has been read.

QByteArray postData;
QFile *file=new QFile("/tmp/image.jpg");
if(!(file->open(QIODevice::ReadOnly))){
    qDebug() << "Could not open file for reading: "<< file->fileName();
    return;
}
//create a header that the server can recognize
postData.insert(0,"--AaB03x\r\nContent-Disposition: form-data; name=\"attachment\"; filename=\"image.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n");
postData.append(file->readAll());
postData.append("\r\n--AaB03x--\r\n");
//here you can add additional parameters that your server may need to parse the data at the end of the url
QString check(QString(POST_URL)+"?fn="+fn+"&md="+md);
QNetworkRequest req(QUrl(check.toLocal8Bit()));
req.setHeader(QNetworkRequest::ContentTypeHeader,"multipart/form-data; boundary=AaB03x");
QVariant l=postData.length();
req.setHeader(QNetworkRequest::ContentLengthHeader,l.toString());
file->close();
//free up memory
delete(file);
//post the data
reply=manager->post(req,postData);
//connect the reply object so we can track the progress of the upload        
connect(reply,SIGNAL(uploadProgress(qint64,qint64)),this,SLOT(updateProgress(qint64,qint64)));

Then the server can access the data like this:

<?php
$filename=$_REQUEST['fn'];
$makedir=$_REQUEST['md'];
if($_FILES["attachment"]["type"]=="image/jpeg"){
if(!move_uploaded_file($_FILES["attachment"]["tmp_name"], "/directory/" . $filename)){
    echo "File Error";
    error_log("Uploaded File Error");
    exit();
};
}else{
print("no file");
error_log("No File");
exit();
}
echo "Success.";
?>

I hope some of this code can help you.

Answer 3

I think the catch is that QNetworkAccessManager does not support chunked transfer encoding when uploading (POST, PUT) data. This means that QNAM must know in advance the length of the data it's going to upload, in order to send the Content-Length header. This implies:

1. either the data does not come from sequential devices, but from random-access devices, which would correctly report their total size through size();
2. or the data comes from a sequential device, but the device has already buffered all of it (this is the meaning of the note about finished()), and will report it (through bytesAvailable(), I suppose);
3. or the data comes from a sequential device which has not buffered all the data, which in turn means
   1. either QNAM reads and buffers itself all the data coming from the device (by reading until EOF)
   2. or the user manually set the Content-Length header for the request.

(About the last two points, see the docs for the QNetworkRequest::DoNotBufferUploadDataAttribute.)

So, QHttpMultiPart somehow shares these limitations, and it's likely that it's choking on case 3. Supposing that you cannot possibly buffer in memory all the data from your "encoder" QIODevice, is there any chance you might know the size of the encoded data in advance and set the content-length on the QHttpPart?

(As a last note, you shouldn't be using QScopedPointer. That will delete the QNR when the smart pointer falls out of scope, but you don't want to do that. You want to delete the QNR when it emits finished()).

Answer 4

From a separate discussion in qt-project and by inspecting the source code it seems that QNAM doesn't work with sequential at all. Both the documentation and code are wrong.