Server/socket programming in C: Data not being sent/received properly?

Question

I am writing a server-side program that listens for connections. When it receives one, it spawns a detached thread and handles the connection. I want it to be able to receive several lines of data being sent from a client, and store it in a data structure (so I can sort it later).

The client does this:

1. Gets the filesize of a .csv file
2. Sends the filesize to the server
3. Opens the .csv
4. Retrieves a line with fgets() and places it into a buffer
5. Sends the number of bytes of this line to the server
6. Sends the data of the line to the server

The goal is to send data over line by line, and have each line stored in a structure over on the server side. The problem is, the data is not being sent over. It seems a single send request gets cut off and is processed in two different recv() calls.

Client Send Loop:

void sendData(int sock, FILE* fd){
    char data[BUFFER];
    int32_t filesize;

    printf("seeking\n");
    if((fseek(fd, 0L, SEEK_END)) == ERROR){
        fprintf(stderr, "An error has occured: %s\n", strerror(errno));
        exit(EXIT_FAILURE);
    };
    if((filesize = ftell(fd)) == ERROR){
        fprintf(stderr, "An error has occured: %s\n", strerror(errno));
        exit(EXIT_FAILURE);
    };
    rewind(fd);

    // subtracting bytes for the col headers 
    //filesize -= COLSIZE;
    fgets(data, sizeof(data), fd);
    printf("strlen of data: %d\n", (int)strlen(data));
    printf("\nfilesize: %ld\n", (long)filesize);
    filesize = filesize - strlen(data);
    printf("\nfilesize: %ld\n", (long)filesize);
    // converting data len to net order
    filesize = htonl(filesize);
    char* len_data = (char*)&filesize;
    // sending filesize
    printf("sending filesize\n");
    send(sock, len_data, sizeof(int32_t), 0);
    filesize = ntohl(filesize);
    printf("\nfilesize: %ld\n", (long)filesize);

    // grabbing first line and sending
    //printf("fgetting test\n");
    while(filesize > 0){
        while(fgets(data, sizeof(data), fd) != NULL){
            if(data[strlen(data) - 1] == '\n'){
                printf("\ntaking newline out\n");
                data[strlen(data) - 1] = '\0';
            }

            // getting and sending size of the line
            int32_t dataSize = sizeof(char)*strlen(data);
            printf("data size of line: %d\n", dataSize);
            dataSize = htonl(dataSize);
            char* lineSize = (char*)&dataSize;
            int j = send(sock, lineSize, sizeof(int32_t), 0);
            printf("size of data sent: %d\n", j);

            // sending the line
            printf("line of data: %s\n", data);
            int i = send(sock, data, sizeof(char)*strlen(data), 0);
            printf("size of data sent: %d\n", i);
            filesize -= i;
            printf("new filesize: %d\n", filesize);
        }
    }

Client Output:

taking newline out
data size of line: 303                                                                                               
size of data sent: 4                                                                                                 
line of data: Color,Andrew Stanton,462,132,475,530,Samantha Morton,640,73058679,Action|Adventure|Sci-Fi,Daryl Sabara,John Carter ,212204,1873,Polly Walker,1,alien|american civil war|male nipple|mars|princess,http://www.imdb.com/title/tt0401729/?ref_=fn_tt_tt_1,738,English,USA,PG-13,263700000,2012,632,6.6,2.35,24000
size of data sent: 303
new filesize: 1001

Server Receive Loop:

    int32_t linesize;
    int32_t filesize;
    // set to 1 to attempt to receive bytes
    receive_int(&filesize,sock_info->sock);
    printf("Recieving file size %d\n", filesize);
    // wait for a recv
    while(filesize){

        printf("\nCurrent File Size: %ld\n",(long) filesize);
        receive_int(&linesize,sock_info->sock);
        printf("Line size: %ld\n", (long)linesize);
        char* filebuf = (char*)malloc(sizeof(char)*linesize);
        printf("size of filebuf: %d\n", sizeof(*filebuf));
        int i = recv(sock_info->sock, filebuf, sizeof(char)*linesize, 0);
        printf("strlen of filebuf: %d\n", strlen(filebuf));
        printf("number of bytes received: %d\n", i);
        printf("Received msg: %s\n",filebuf);
        filesize -= i;


    }

Server Output

Current File Size: 1304                                                                                              
Line size: 303                                                                                                       
size of filebuf: 1                                                                                                   
strlen of filebuf: 165                                                                                               
number of bytes received: 165                                                                                        
Received msg: Color,Andrew Stanton,462,132,475,530,Samantha Morton,640,73058679,Action|Adventure|Sci-Fi,Daryl Sabara,John Carter ,212204,1873,Polly Walker,1,alien|american civil                          

Current File Size: 1139                                                                                              
Line size: 2002875004                                                                                                
size of filebuf: 1                                                                                                   
strlen of filebuf: 134                                                                                               
number of bytes received: 1139
Received msg: male nipple|mars|princess,http://www.imdb.com/title/tt0401729/?ref_=fn_tt_tt_1,738,English,USA,PG-13,263700000,2012,632,6.6,2.35,24000

Answer 1

As indicated in the man page for recv:

The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested.

As indicated in the man page for send:

With a zero flags argument, send() is equivalent to write(2). ... On success, these calls return the number of bytes sent.

As indicated in the man page for write:

writes up to count bytes from the buffer starting at...

As you can see, network send and recv offer no guarantees as to the length of the data actually sent or the length of the data received in each function call.

Moreover, it is very common, as packets move through the network, for a single packet to be fragmented into a number of packets (for example, due to differences in network MTU settings).

The only way to treat network data is as a continuous stream.

In order to find new-line markers, the receiving endpoint will have to concat any received data and cut it up along the new-line markers.

I recommend Beej's guide as a good resource to start up on network coding.

Good Luck.