I've searched StackOverflow trying to find a similar problem, but haven't come across it, so I am posting this question.
I am trying to write an C++ HTTPS client using Microsoft's SChannel libraries, and I'm getting stochastic errors with chunked message transfer. This issue only seems to occur on very long downloads -- short ones generally work OK. Most of the time the code works properly -- even for long downloads -- but occasionally the recv() command gracefully timesout, disconnecting my TLS session, and other times, I get an incomplete last packet. The stochastic errors appear to be the result of the different size chunks and encryption blocks the server is using to pass the data. I know I need to handle this variation, but while this would be easy to solve on an unencrypted HTTP connection, the encryption aspect is causing me problems.
First, the timeout problem, which occurs about 5% of the time I request large HTTP requests (about 10 MB of data from a single HTTP GET request).
The timeout is resulting because on the last chunk I have specified a bigger receive buffer than the data remaining on a blocking socket. The obvious fix to this is to only request exactly the number of bytes I need for the next chunk, and that is what I did. But for some reason, the amount received from each request is less than what I request, yet appears to be missing no data after decryption. I'm guessing this must be due to some compression in the data stream, but I don't know. IN any event, if it is using compression, I have no idea how to translate the size of the decrypted uncompressed byte stream into the size of compressed encrypted byte stream including the encryption headers and trailers to request the exact right number of bytes. Can anyone help me do that?
The alternative approach is for me to just look for two CR+LFs in a row, which would also signal the end of the HTTPS response. But because the data is encrypted, I can't figure out how to look byte by byte. SChannel's DecryptMessage() seems to do its decryptions in blocks, not byte by byte. Can anyone in this forum provide any advice on how to do byte-by-byte decryption to enable me to look for the end of the chunked output?
The second problem is DecryptMessage sometimes erroneously thinks it is done decrypting before I reach the actual end of the message. The resultant behavior is I go on to the next HTTP request, and I get the rest of the previous response where I am expecting to see the header of the new request.
The obvious solution to this is to check the contents of the decrypted message to see if we actually reached the end, and if not, try to receive more data before sending the next HTTP request. But when I do this, and try to decrypt, I get a decryption error message.
Any advice/help anyone can provide on a strategies would be appreciated. I've attached the relevant code sections for the read/decrypt process of the HTTP body -- I'm not including the header read and parsing because that is working without any problems.
do
{
// Note this receives large files OK, but I can't tell when I hit the end of the buffer, and this
// hangs. Need to consider a non-blocking socket?
// numBytesReceived = recv(windowsSocket, (char*)inputBuffer, inputBufSize, 0);
m_ErrorLog << "Next read size expected " << nextReadSize << endl;
numBytesReceived = recv(windowsSocket, (char*)inputBuffer, nextReadSize, 0);
m_ErrorLog << "NumBytesReceived = " << numBytesReceived << endl;
if (m_BinaryBufLen + numBytesReceived > m_BinaryBufAllocatedSize)
::EnlargeBinaryBuffer(m_BinaryBuffer,m_BinaryBufAllocatedSize,m_BinaryBufLen,numBytesReceived+1);
memcpy(m_BinaryBuffer+m_BinaryBufLen,inputBuffer,numBytesReceived);
m_BinaryBufLen += numBytesReceived;
lenStartDecryptedChunk = decryptedBodyLen;
do
{
// Decrypt the received data.
Buffers[0].pvBuffer = m_BinaryBuffer;
Buffers[0].cbBuffer = m_BinaryBufLen;
Buffers[0].BufferType = SECBUFFER_DATA; // Initial Type of the buffer 1
Buffers[1].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 2
Buffers[2].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 3
Buffers[3].BufferType = SECBUFFER_EMPTY; // Initial Type of the buffer 4
Message.ulVersion = SECBUFFER_VERSION; // Version number
Message.cBuffers = 4; // Number of buffers - must contain four SecBuffer structures.
Message.pBuffers = Buffers; // Pointer to array of buffers
scRet = m_pSSPI->DecryptMessage(phContext, &Message, 0, NULL);
if (scRet == SEC_E_INCOMPLETE_MESSAGE)
break;
if( scRet == SEC_I_CONTEXT_EXPIRED )
{
m_ErrorLog << "Server shut down connection before I finished reading" << endl;
m_ErrorLog << "# of Bytes Requested = " << nextReadSize << endl;
m_ErrorLog << "# of Bytes received = " << numBytesReceived << endl;
m_ErrorLog << "Decrypted data to this point = " << endl;
m_ErrorLog << decryptedBody << endl;
m_ErrorLog << "BinaryData just decrypted: " << endl;
m_ErrorLog << Buffers[0].pvBuffer << endl;
break; // Server signalled end of session
}
if( scRet != SEC_E_OK &&
scRet != SEC_I_RENEGOTIATE &&
scRet != SEC_I_CONTEXT_EXPIRED )
{
DisplaySECError((DWORD)scRet,errmsg);
m_ErrorLog << "CSISPDoc::ReadDecrypt(): " << "Failed to decrypt message--Error=" << errmsg;
if (decryptedBody)
m_ErrorLog << decryptedBody << endl;
return scRet;
}
// Locate data and (optional) extra buffers.
pDataBuffer = NULL;
pExtraBuffer = NULL;
for(i = 1; i < 4; i++)
{
if( pDataBuffer == NULL && Buffers[i].BufferType == SECBUFFER_DATA )
pDataBuffer = &Buffers[i];
if( pExtraBuffer == NULL && Buffers[i].BufferType == SECBUFFER_EXTRA )
pExtraBuffer = &Buffers[i];
}
// Display the decrypted data.
if(pDataBuffer)
{
length = pDataBuffer->cbBuffer;
if( length ) // check if last two chars are CR LF
{
buff = (PBYTE)pDataBuffer->pvBuffer; // printf( "n-2= %d, n-1= %d \n", buff[length-2], buff[length-1] );
if (decryptedBodyLen+length+1 > decryptedBodyAllocatedSize)
::EnlargeBuffer(decryptedBody,decryptedBodyAllocatedSize,decryptedBodyLen,length+1);
memcpy_s(decryptedBody+decryptedBodyLen,decryptedBodyAllocatedSize-decryptedBodyLen,buff,length);
decryptedBodyLen += length;
m_ErrorLog << buff << endl;
}
}
// Move any "extra" data to the input buffer -- this has not yet been decrypted.
if(pExtraBuffer)
{
MoveMemory(m_BinaryBuffer, pExtraBuffer->pvBuffer, pExtraBuffer->cbBuffer);
m_BinaryBufLen = pExtraBuffer->cbBuffer; // printf("inputStrLen= %d \n", inputStrLen);
}
}
while (pExtraBuffer);
if (decryptedBody)
{
if (incompletePacket)
p1 = decryptedBody + lenStartFragmentedPacket;
else
p1 = decryptedBody + lenStartDecryptedChunk;
p2 = p1;
pEndDecryptedBody = decryptedBody+decryptedBodyLen;
if (lastDecryptRes != SEC_E_INCOMPLETE_MESSAGE)
chunkSizeBlock = true;
do
{
while (p2 < pEndDecryptedBody && (*p2 != '\r' || *(p2+1) != '\n'))
p2++;
// if we're here, we probably found the end of the current line. The pattern we are
// reading is chunk length, chunk, chunk length, chunk,...,chunk lenth (==0)
if (*p2 == '\r' && *(p2+1) == '\n') // new line character -- found chunk size
{
if (chunkSizeBlock) // reading the size of the chunk
{
pStartHexNum = SkipWhiteSpace(p1,p2);
pEndHexNum = SkipWhiteSpaceBackwards(p1,p2);
chunkSize = HexCharToInt(pStartHexNum,pEndHexNum);
p2 += 2; // skip past the newline character
chunkSizeBlock = false;
if (!chunkSize) // chunk size of 0 means we're done
{
bulkReadDone = true;
p2 += 2; // skip past the final CR+LF
}
nextReadSize = chunkSize+8; // chunk + CR/LF + next chunk size (4 hex digits) + CR/LF + encryption header/trailer
}
else // copy the actual chunk
{
if (p2-p1 != chunkSize)
{
m_ErrorLog << "Warning: Actual chunk size of " << p2 - p1 << " != stated chunk size = " << chunkSize << endl;
}
else
{
// copy over the actual chunk data //
if (m_HTTPBodyLen + chunkSize > m_HTTPBodyAllocatedSize)
::EnlargeBuffer(m_HTTPBody,m_HTTPBodyAllocatedSize,m_HTTPBodyLen,chunkSize+1);
memcpy_s(m_HTTPBody+m_HTTPBodyLen,m_HTTPBodyAllocatedSize,p1,chunkSize);
m_HTTPBodyLen += chunkSize;
m_HTTPBody[m_HTTPBodyLen] = 0; // null-terminate
p2 += 2; // skip over chunk and end of line characters
chunkSizeBlock = true;
chunkSize = 0;
incompletePacket = false;
lenStartFragmentedPacket = 0;
}
}
p1 = p2; // move to start of next chunk field
}
else // got to end of encrypted body with no CR+LF found --> fragmeneted chunk. So we need to read and decrypt at least one more chunk
{
incompletePacket = true;
lenStartFragmentedPacket = p1-decryptedBody;
}
}
while (p2 < pEndDecryptedBody);
lastDecryptRes = scRet;
}
}
while (scRet == SEC_E_INCOMPLETE_MESSAGE && !bulkReadDone);