From a Swift 2.1-based iOS client app using AFNetworking 2.0, I'm uploading a PNG image file to a Node.js server. The issue I'm running into is that when I use a .png file name extension (as is usual), the file gets larger when it is uploaded. The original file size is: 917,630 bytes, and the uploaded size is 1,298,016 bytes. Curiously, this not a completely corrupting change in the file contents. I.e., I can still view the image with Preview on Mac OS X (though see Update1 below).
Here's the guts of of my client app upload code:
public class func uploadFileTo(serverURL: NSURL, fileToUpload:NSURL, withParameters parameters:[String:AnyObject]?, completion:((serverResponse:[String:AnyObject]?, error:NSError?)->())?) {
Log.special("serverURL: \(serverURL)")
var sendParameters:[String:AnyObject]? = parameters
#if DEBUG
if (SMTest.session.serverDebugTest != nil) {
if parameters == nil {
sendParameters = [String:AnyObject]()
}
sendParameters![SMServerConstants.debugTestCaseKey] = SMTest.session.serverDebugTest
}
#endif
self.manager.POST(serverURL.absoluteString, parameters: sendParameters, constructingBodyWithBlock: { (formData: AFMultipartFormData) in
// NOTE!!! the name: given here *must* match up with that used on the server in the "multer" single parameter.
// Was getting an odd try/catch error here, so this is the reason for "try!"; see https://github.com/AFNetworking/AFNetworking/issues/3005
// 12/12/15; I think this issue was because I wasn't doing the do/try/catch, however.
do {
try formData.appendPartWithFileURL(fileToUpload, name: SMServerConstants.fileUploadFieldName)
} catch let error {
let message = "Failed to appendPartWithFileURL: \(fileToUpload); error: \(error)!"
Log.error(message)
completion?(serverResponse: nil, error: Error.Create(message))
}
}, success: { (request: AFHTTPRequestOperation, response:AnyObject) in
if let responseDict = response as? [String:AnyObject] {
print("AFNetworking Success: \(response)")
completion?(serverResponse: responseDict, error: nil)
}
else {
let error = Error.Create("No dictionary given in response")
print("**** AFNetworking FAILURE: \(error)")
completion?(serverResponse: nil, error: error)
}
}, failure: { (request: AFHTTPRequestOperation?, error:NSError) in
print("**** AFNetworking FAILURE: \(error)")
completion?(serverResponse: nil, error: error)
})
On the Node.js, here's the package.json:
{
"name": "node1",
"version": "1.0.0",
"description": "Test",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"body-parser": "^1.14.1",
"fs-extra": "^0.26.2",
"google-auth-library": "^0.9.7",
"googleapis": "^2.1.6",
"mongodb": "^2.0.49",
"multer": "^1.1.0",
"sweet": "^0.1.1",
"tracer": "^0.8.2"
},
"devDependencies": {
"sweet.js": "^0.7.4"
}
}
Here's the initial part of my server "index.js" file (a lot of which is specific to my project):
// Before the files get moved to their specific-user destination.
const initialUploadDirectory = './initialUploads/';
// TODO: What is safe mode in mongo? E.g., see https://mongodb.github.io/node-mongodb-native/api-generated/collection.html#insert
// See also options on insert https://mongodb.github.io/node-mongodb-native/api-generated/collection.html#insert
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
// https://github.com/expressjs/multer
var multer = require('multer');
var fse = require('fs-extra');
// Local modules.
var ServerConstants = require('./ServerConstants');
var Mongo = require('./Mongo');
var Operation = require('./Operation');
var PSLock = require('./PSLock');
var PSOutboundFileChange = require('./PSOutboundFileChange.sjs');
var FileTransfers = require('./FileTransfers');
var File = require('./File.sjs')
var logger = require('./Logger');
var PSOperationId = require('./PSOperationId.sjs');
var PSFileIndex = require('./PSFileIndex');
// See http://stackoverflow.com/questions/31496100/cannot-app-usemulter-requires-middleware-function-error
// See also https://codeforgeek.com/2014/11/file-uploads-using-node-js/
// TODO: Limit the size of the uploaded file.
// TODO: Is there a way with multer to add a callback that gets called periodically as an upload is occurring? We could use this to "refresh" an activity state for a lock to make sure that, even with a long-running upload (or download) if it is still making progress, that we wouldn't lose a lock.
var upload = multer({ dest: initialUploadDirectory}).single(ServerConstants.fileUploadFieldName)
// http://stackoverflow.com/questions/4295782/how-do-you-extract-post-data-in-node-js
app.use(bodyParser.json({extended : true}));
And here's the initial (relevant) part of the REST/API entry point for the upload:
app.post('/' + ServerConstants.operationUploadFile, upload, function (request, response) {
var op = new Operation(request, response);
if (op.error) {
op.end();
return;
}
/* request.file has the info on the uploaded file: e.g.,
{ fieldname: 'file',
originalname: 'upload.txt',
encoding: '7bit',
mimetype: 'text/plain',
destination: './uploads/',
filename: 'e9a4080c46777d6341518afedec8af31',
path: 'uploads/e9a4080c46777d6341518afedec8af31',
size: 22 }
*/
op.validateUser(function (psLock, psOperationId) {
// User is on the system.
//console.log("request.file: " + JSON.stringify(request.file));
// Make sure user/device has started uploads. i.e., make sure this user/device has the lock.
if (!psLock) {
var message = "Error: Don't have the lock!";
logger.error(message);
op.endWithRCAndErrorDetails(ServerConstants.rcServerAPIError, message);
} else if (psOperationId.operationStatus ==
ServerConstants.rcOperationStatusInProgress) {
// This check is to deal with error recovery.
var message = "Error: Have lock, but operation is already in progress!";
logger.error(message);
op.endWithRCAndErrorDetails(ServerConstants.rcServerAPIError, message);
} else {
logger.info("We've got the lock!");
// Leave the parameter checking below until after checking for the lock because we're just checking for a lock, and not creating a lock.
// 12/12/15; Ran into a bug where the upload failed, and .file object wasn't defined.
if (!isDefined(request.file) || !isDefined(request.file.path)) {
var message = "No file uploaded!";
logger.error(message);
op.endWithRCAndErrorDetails(ServerConstants.rcServerAPIError, message);
return;
}
logger.info(JSON.stringify(request.file));
...
And here's the logger.info output:
{"fieldname":"file","originalname":"Test.png","encoding":"7bit","mimetype":"image/png","destination":"./initialUploads/","filename":"da17e16904ed376fb21052c80b88da12","path":"initialUploads/da17e16904ed376fb21052c80b88da12","size":1298016}
When I change the file extension to .bin (just some non-standard extension), the file size is not increased-- i.e., it remains the smaller value I'd originally expected.
The ratio between the two files sizes is 1.41453091 (= 1,298,016/917,630). Which looks oddly close to the square root of 2. Some encoding issue?
Thoughts?
Update1: When I use ImageMagick's identify program, I get reasonable output for the image before uploading, but after uploading (with the larger image), I get:
$ identify -verbose example2.png identify: CgBI: unhandled critical chunk
example2.png' @ error/png.c/MagickPNGErrorHandler/1630. identify: corrupt imageexample2.png' @ error/png.c/ReadPNGImage/3959.
Update2: I think I can now say for sure that this is a client-side issue related to AFNetworking and not a server-side issue related to Node.js. I make this inference because when I make a simplified Node.js server (using all of the same "header" code as my actual server) as below:
index.js
'use strict';
require('sweet.js').loadMacro('./macros.sjs');
var server = require("./Server.sjs");
Server.sjs
// Before the files get moved to their specific-user destination.
const initialUploadDirectory = './initialUploads/';
// TODO: What is safe mode in mongo? E.g., see https://mongodb.github.io/node-mongodb-native/api-generated/collection.html#insert
// See also options on insert https://mongodb.github.io/node-mongodb-native/api-generated/collection.html#insert
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
// https://github.com/expressjs/multer
var multer = require('multer');
var fse = require('fs-extra');
// Local modules.
var ServerConstants = require('./ServerConstants');
var Mongo = require('./Mongo');
var Operation = require('./Operation');
var PSLock = require('./PSLock');
var PSOutboundFileChange = require('./PSOutboundFileChange.sjs');
var FileTransfers = require('./FileTransfers');
var File = require('./File.sjs')
var logger = require('./Logger');
var PSOperationId = require('./PSOperationId.sjs');
var PSFileIndex = require('./PSFileIndex');
// See http://stackoverflow.com/questions/31496100/cannot-app-usemulter-requires-middleware-function-error
// See also https://codeforgeek.com/2014/11/file-uploads-using-node-js/
// TODO: Limit the size of the uploaded file.
// TODO: Is there a way with multer to add a callback that gets called periodically as an upload is occurring? We could use this to "refresh" an activity state for a lock to make sure that, even with a long-running upload (or download) if it is still making progress, that we wouldn't lose a lock.
var upload = multer({ dest: initialUploadDirectory}).single(ServerConstants.fileUploadFieldName)
// http://stackoverflow.com/questions/4295782/how-do-you-extract-post-data-in-node-js
app.use(bodyParser.json({extended : true}));
// Server main.
Mongo.connect();
app.post('/upload', upload, function (request, response) {
console.log(JSON.stringify(request.file));
var result = {};
response.end(JSON.stringify(result));
});
app.listen(8081);
console.log('Server running at http://127.0.0.1:8081/');
and then test that using Postman, uploading my example PNG file, I get the uploaded file with no increase in size. Here's the Node.js output:
{"fieldname":"file","originalname":"Example.png","encoding":"7bit","mimetype":"image/png","destination":"./initialUploads/","filename":"ac7c8c93d50bf48cf6042409ef990658","path":"initialUploads/ac7c8c93d50bf48cf6042409ef990658","size":917630}
Then, when I drop the above app.post method into my actual server, and again test the upload with Postman (not my example app using AFNetworking), I still do not get the increase in file size:
{"fieldname":"file","originalname":"Example.png","encoding":"7bit","mimetype":"image/png","destination":"./initialUploads/","filename":"a40c738a172eb9ea6cccce357338beeb","path":"initialUploads/a40c738a172eb9ea6cccce357338beeb","size":917630}
So far so good, without using AFNetworking.
And finally, when I add an additional test into my iOS client app, using the simplified app.post on the server, and using AFNetworking for the client (and I'm using AFNetworking 3 now), but only using the file upload post on the client, I get:
{"fieldname":"file","originalname":"Example.png","encoding":"7bit","mimetype":"image/png","destination":"./initialUploads/","filename":"8c1116337fd2650d4f113b227252e555","path":"initialUploads/8c1116337fd2650d4f113b227252e555","size":1298016}
That is, using AFNetworking again on the client, I again get the larger file size.
