2
votes
  1. Is it possible to access the data as it's coming in with HTML5 using the FileReader API and the onprogress event?

  2. If so, is there an "online" version of MD5 or other fast hashing algorithm so that I can begin computing the hash before the file is fully read?

I would like to compute hashes client-side and send just the hash to the server before sending an entire file to check for duplicates before initiating a file upload.

I am not concerned with support for older browsers at this time.

Edit: I recognize that a hash collision does not guarantee a duplicate file, and the only way to be sure is to check byte-by-byte which would mean I would have to upload the file anyway. The probability is low enough that I'm willing to take this risk; worst case I prompt the user and say "This file already appears to be on the server; are you sure you want to upload it?"

2

2 Answers

2
votes

is there an "online" version of MD5 or other fast hashing algorithm so that I can begin computing the hash before the file is fully read?

Yes, you can use sjcl if you want to use SHA. sjcl has no native support for MD5, so you'll have to write it yourself (though I'm sure someone else has done it already). CryptoJS has native MD5 support but is significantly slower.

I recognize that a hash collision does not guarantee a duplicate file [...] The probability is low enough that I'm willing to take this risk;

The probability is low enough that there's better chance of a meteor hitting the Earth and ending human life (thus removing the need for hashing altogether) than for a collision to naturally occur. Unless the user crafted a collision on purpose, of course, since MD5's collision resistance is broken.

Here's a live demo of what I believe you're trying to accomplish, minus the "access data as it comes" part. I am not sure if that's possible. I wrote this a long time ago and it uses CryptoJS, so performance isn't that great but it gets the job done. The important chunks are:

function handleFileSelect(evt) 
{
    evt.stopPropagation();
    evt.preventDefault();

    var files = evt.target.files || evt.dataTransfer.files; // FileList object.

    for (var i=0, file; file = files[i]; ++i)
    {
        // this creates the FileReader and reads stuff as text
        var fr = new FileReader();

        fr.onload = (function(theFile) {
            return function (e) {
                var hashes = parsePseudoBuffer(e.target.result);

                document.getElementById('output').innerHTML += '<br />' + theFile.name + '<br />' 
                + 'MD5: ' + hashes.md5 + '<br />' + 'SHA1: ' + hashes.sha1 + '<br />' ;

            };
        }) (file);

        fr.readAsArrayBuffer(file); // ArrayBuffer
    }

}

function parsePseudoBuffer(result)
{

    var buffs = new Uint8Array(result); // buffer thingie       
    var md5 = CryptoJS.algo.MD5.create();
    var sha1 = CryptoJS.algo.SHA1.create();     
    var bufsize = 8 * 1024; // 8K buffer

    for (var bstart=0, bend=bufsize; bstart < buffs.length; bstart+=bufsize, bend+= bufsize)
    {
        var data = CryptoJS.lib.WordArray.create(buffs.subarray(bstart, bend)); 
        md5.update(data);
        sha1.update(data);          
    }

    md5 = md5.finalize(); 
    sha1 = sha1.finalize();         

    return {'md5': md5, 'sha1': sha1} ;

}
2
votes

I did some experimenting. It looks like we can get the last chunk read inside the onprogress event by utilizing the incomplete result on the reader object. It only appears to be accessible if we use reader.readAsArrayBuffer (Chrome only?) or reader.readAsBinaryString. The problem with strings is that if that if you want to take a chunk of it, you have to slice it which makes a copy (very slow).

ArrayBuffers have a .subarray method which creates a view into the buffer, without copying any data. This is exactly what we want. However, it doesn't appear to be available on the base class; and it's not clear from the documentation what happens when we construct a derived class (e.g. Uint8Array) using this buffer, but considering the original buffer is accessible via a readonly property, I'm assuming it's not copying.

Both sjcl and CryptoJS conveniently have .update methods which will take in this ArrayBufferView so that you can update your hash on the fly. Thus, I have come up with the following solution (using jQuery, underscore and sjcl):

$(document).on('drop', function(dropEvent) {
    dropEvent.preventDefault();

    _.each(dropEvent.originalEvent.dataTransfer.files, function(file) {
        var reader = new FileReader();
        var pos = 0;

        var hash = new sjcl.hash.sha256();

        reader.onprogress = function(progress) {
            var chunk = new Uint8Array(reader.result, pos, progress.loaded - pos);
            pos = progress.loaded;
            hash.update(chunk);
            if(progress.lengthComputable) {
                console.log((progress.loaded/progress.total*100).toFixed(1)+'%');
            }
        };

        reader.onload = function() {
            var chunk = new Uint8Array(reader.result, pos);
            if(chunk.length > 0) hash.update(chunk);
            console.log(sjcl.codec.hex.fromBits(hash.finalize()));
        };

        reader.readAsArrayBuffer(file);
    });
});

Note that this solution presently only works in Chrome and is fairly slow. I think sjcl isn't just hashing the file, but it's key-strengthening it, which really isn't what I want. Will investigate more later.