Java Implement byte range serving without servlets using only Java api

2
votes

I have been writing a http server in Java for quite some time(almost two years now?) and I am still having issues getting byte range requests to work. I am only using the socket's input and output stream for raw byte data transfer(i.e. file downloads/uploads), and a PrintWriter for sending response headers/strings to connecting clients(such as HTTP/1.1 200 OK and so forth).

I do not wish to use any servlets or apis(such as HTTPURLConnection or whatever). I want to do it "vanilla style".

I am able to serve normal web pages quite nicely and quickly(such as file browsing, uploads and downloads, watching movies, listening to music, viewing pdf files, text, pictures, gif files, etc.), so that is not an issue.

However, whenever I try implementing byte range requests, the server receives the client's request perfectly, parses the given data, prepares the file input stream for sending, and then when I send the data to the client, the client always drops the connection with software caused connection abort: socket write error.(I need byte range requests for cases such as: watching an hour long video, then the dang wifi signal drops out and you don't want to re-watch the entire video from square one, resuming a 'paused download', etc.)

This really has me stumped, and I have indeed searched for java examples of serving byte range requests, all of which fail when I try to implement them. I have even tried starting from scratch and testing that, and it produces the same result. Here are the code snippets relevant to what I am trying to accomplish:

Sending and receiving normal web pages(works fine, this is an example):

    ...
        OutputStream outStream = client.getOutputStream();
        PrintWriter out = new PrintWriter(new OutputStreamWriter(outStream, StandardCharsets.UTF_8), true);
        out.println("HTTP/1.1 200 OK");
        out.println("Content-Type: text/html; charset=\"UTF-8\"");
        String responseStr = "<!DOCTYPE html><html><head><title>Hello, world!</title></head><body><string>This is a simple example webpage!</string></body></html>";
        if(acceptEncoding.contains("gzip") && responseStr.length() > 33) {
            out.println("Content-Encoding: gzip");
            byte[] r = StringUtil.compressString(responseStr, "UTF-8");
            out.println("Content-Length: " + r.length);
            out.println("");
            if(protocol.equalsIgnoreCase("GET")) {
                outStream.write(r);
            }
        } else {
            out.println("Content-Length: " + responseStr.length());
            out.println("");
            if(request.protocol.equalsIgnoreCase("GET")) {
                out.println(responseStr);
            }
        }
        out.flush();
        //followed by closing resources, etc.
    ...

Byte range serving(after client request data is parsed):

public static final long copyInputStreamToOutputStream(InputStream input, OutputStream output, long startBytes, long endBytes) throws IOException {
    byte[] buffer = new byte[20480];//1024
    long count = 0;
    int read;

    input.skip(startBytes);
    long toRead = (endBytes - startBytes) + 1;

    while((read = input.read(buffer)) > 0) {
        if((toRead -= read) > 0) {
            output.write(buffer, 0, read);//Socket error happens on this line mostly
            output.flush();
        } else {
            output.write(buffer, 0, (int) toRead + read);//but also on this line sometimes
            output.flush();
            break;
        }
        count += read;
    }
    return count;
}

For anyone that is interested, the server running on this basic code is online at redsandbox.no-ip.org (points to my server), and at the moment I have byte requests disabled with Accept-Ranges: none instead of Accept-Ranges: bytes, but I can turn it on again to test it.

If I need to add more code in, please let me know! Thank you for your time. Alternately, if you wish to view my server's code in full, you can view it on github: https://github.com/br45entei/JavaWebServer

2
javahttpserverrangebyte
I like such questions, cause I believe too that it's possible to rewrite standard lib's code in more efficient way for particular list of requirements and/or limitations (especially when your program has to survive under high load or serve needs of high throughput data flows). - rsutormin
Do you have your code commited to some github repo? - rsutormin
Unfortunately no, but I can attempt to do so real quick if you would like. - Brian_Entei
It could be easier to deal with all the code. Of course if you don't mind to make your code open source. - rsutormin
Inputstream.skip(long) is not guaranteed to skip n bytes by the javadoc, check for return value. But I think local fileinputstream is rather consistent. Also make sure none previous stackcall functions use integer, it may overflow. Debug print start+end+skipped values to be sure. Btw your style of code is fine, no problems understanding it. - Whome

2 Answers

2
votes

Would this read-write loop work fine, sure looks streamlined? This is out of my head so minor syntax errors may be found. I am expecting startIdx+endIdx both are inclusive indicies.

public static final long copyInputToOutput(InputStream input, OutputStream output, long startIdx, long endIdx) throws IOException {
    final long maxread = 24*1024;
    byte[] buffer = new byte[(int)maxread];
    input.skip(startIdx);
    long written=0;
    long remaining = endIdx-startIdx+1;
    while(remaining>0) {
        int read = input.read(buffer, 0, (int)Math.min(maxread, remaining));
        output.write(buffer, 0, read);
        remaining -= read;
        written += read;
    }
    output.flush();
    return written;
}
1
votes

So I figured out what my issues were, thanks in part to Whome. Firstly, I wasn't sending the correct Content-Length: header to the client, and secondly, my endbytes variable was being calculated incorrectly. Here is the working code:

public static final long copyInputStreamToOutputStream(InputStream input, OutputStream output, long startBytes, long endBytes, final long contentLength, boolean debug) throws IOException {
    byte[] buffer = new byte[20480];//1024
    long count = 0;
    int read;
    long skipped = input.skip(startBytes);//I tested this quite a few times with different files and it does indeed skip the desired amount of bytes.
    final long length = (endBytes - startBytes) + 1;
    if(debug) {
        println("Start bytes: " + startBytes + "; End bytes: " + endBytes + "; Skipped bytes: " + skipped + "; Skipped equals startBytes: " + (skipped == startBytes) + "; Length: " + length + "; Total file size: " + contentLength);
    }
    long toRead = length;
    while((read = input.read(buffer)) != -1) {
        count += read;
        long bytesLeft = length - count;
        if(debug) {
            println("read: " + read + "; toRead: " + toRead + "; length: " + length + "; count: " + count + "; bytesLeft: " + bytesLeft);
        }
        toRead -= read;
        if(bytesLeft >= buffer.length) {//Changed the if statement here so that we actually send the last amount of data to the client instead of whatever is leftover in the buffer
            output.write(buffer, 0, read);
            output.flush();
        } else {
            output.write(buffer, 0, read);//Send the currently read bytes
            output.flush();
            bytesLeft = length - count;//recalculate the remaining byte count
            read = input.read(buffer, 0, (int) bytesLeft);//read the rest of the required bytes
            count += read;
            if(debug) {
                println("[Last one!]read: " + read + "; toRead: " + toRead + "; length: " + length + "; count: " + count + "; bytesLeft: " + bytesLeft);
            }
            output.write(buffer, 0, read);//and send them over
            output.flush();
            break;
        }
    }
    return count;
}

Since the endBytes variable in my case was one more than it needed to be, I was trying compensate by adjusting the code around it. I needed to simply subtract one from it, however. The if statement with the buffer.length is to ensure that the last of the bytes are sent to the client. In my tests without it, the client(google chrome) hangs and waits for the remaining bytes, but never receives them, then when the 30 second time out closes the connection, the webpage resets. I have yet to test this with other browsers, but I think it should work.