MongoError: key $ must not start with '$' when store JSON object generated by xml2js module

5
votes

Everyone, 

MongoDB db.version() is 3.0.5
mongodb package.json shows 2.0.42 version
xml2js package.json shows 0.4.9 version

I have already googled the error, and read through all the existing questions and none seem to match my circumstances (neither the jira issue, nor the google group discussions, etc). This could be related to mongodb native driver (or MongoDB server version), xml2js or something else.

I have done lots of testing and did found a way round the problem but, I am curious to know what is the issue.

I have an app that does the following:

  1. Upload XForm (survey.xml) to the ExpressJS
  2. Use the xml2js library to convert the XML to JSON object (sample shown below)
  3. Iterate through JSON object created in 2 and remove unnecessary fields (snippet code shown below)
  4. Push the modified JSON object in 3 to MongoDB using mongodb native driver

Step 4 fails with error (shown in title of the question).

  1. There are 2 JSON objects, one is the survey form, containing $ keys but, before I insert the document into mongodb, I remove the $ keys using the following code

    (function traverse(o) {
for (var i in o) {
  if (o[i] !== null && typeof(o[i])=="object") {
      //going on step down in the object tree!!
    if(o[i].$) {
      var ref = "";
      if(o[i].$.ref) {
        ref = o[i].$.ref;
      } else if (o[i].$.nodeset) {
        ref = o[i].$.nodeset;
      }
      o[i].ref = ref;
      o[i].$ = undefined;
      var chunks = ref.split('/');
      o[i]['name'] = chunks[chunks.length - 1];
    }
    traverse(o[i]);
  }
}
})(body);

  2. I console log the output of the above before adding it to MongoDB and the console.log shows no sign of keys with $.

  3. Assuming that step 2 actually fails due to a $ in the key, which is not true, I have another JSON which is the response of the survey and contains no $ whatsoever, but that also fails with the same error (MongoError: key $ must not start with '$'). 

    db.collection('submissions').insert(jsonObject, function(err, result) {
  if(err) console.log('error is : ' + err);
  console.log('insertion result : ' + JSON.stringify(result));
});

Below is the XML form submission 

   <?xml version='1.0' ?>
   <ppe id="ppe_checklist_new">
    <starting_repeat>
        <location>dark_room</location>
        <ppe_dark>safety_glasses</ppe_dark>
        <xyz_group>
            <condition>condition_missing</condition>
            <test_condition>b</test_condition>
        </xyz_group>
    </starting_repeat>
    <starting_repeat>
        <location>nitrogen_store</location>
        <ppe_nitrogen>leather_gloves_s</ppe_nitrogen>
        <xyz_group>
            <condition>condition_replacing</condition>
            <test_condition />
        </xyz_group>
    </starting_repeat>
    <starting_repeat>
        <location>nitrogen_store</location>
        <ppe_nitrogen>blue_gloves_m</ppe_nitrogen>
        <xyz_group>
            <condition />
            <test_condition>b</test_condition>
        </xyz_group>
    </starting_repeat>
    <starting_repeat>
        <location>cold_room_first</location>
        <ppe_cold>hearing_muff_1</ppe_cold>
        <xyz_group>
            <condition>condition_ok</condition>
            <test_condition>f</test_condition>
        </xyz_group>
    </starting_repeat>
    <sample_group>
        <date>2015-08-24</date>
        <random_number>55</random_number>
    </sample_group>
    <another_group>
        <another_repeat>
            <sample_text>Sample text 1</sample_text>
            <image />
        </another_repeat>
        <another_repeat>
            <sample_text>Sample text 2</sample_text>
            <image />
        </another_repeat>
    </another_group>
    <form_done>OK</form_done>
    <survey_start>2015-08-24T16:55:23.185+01</survey_start>
    <survey_end>2015-08-24T16:57:24.460+01</survey_end>
    <survey_day>2015-08-24</survey_day>
    <survey_device>353490061313389</survey_device>
    <meta>
        <instanceID>uuid:2aba0eff-5350-47e3-9e9c-9606d2c9e7d6</instanceID>
    </meta>
</ppe>

I feed the above to xml2js module with following configs: 

function parseXMLToJS(filename, path, callback) {
  var parser = new xml2js.Parser({explicitArray:false});
  var fileURL = path + filename;

  var data = fs.readFile(fileURL, function(err, data) {
    if(err) {
      logger.error('Error reading submission file. Error %', err);
    } else {
      parser.parseString(data, function (err, result) {
        if(err) {
          logger.error('Error parsing XML to JS. Error : %',  err);
          callback({parsed:false, result:result});
        } else {
          callback({parsed:true, result:result});
        }
      });
    }
  });
}

and the module generates a JSON object as shown below: 

{
    "ppe": {
        "starting_repeat": [
            {
                "location": "dark_room",
                "ppe_dark": "safety_glasses",
                "xyz_group": {
                    "condition": "condition_missing",
                    "test_condition": "b"
                }
            },
            {
                "location": "nitrogen_store",
                "ppe_nitrogen": "leather_gloves_s",
                "xyz_group": {
                    "condition": "condition_replacing",
                    "test_condition": ""
                }
            },
            {
                "location": "nitrogen_store",
                "ppe_nitrogen": "blue_gloves_m",
                "xyz_group": {
                    "condition": "",
                    "test_condition": "b"
                }
            },
            {
                "location": "cold_room_first",
                "ppe_cold": "hearing_muff_1",
                "xyz_group": {
                    "condition": "condition_ok",
                    "test_condition": "f"
                }
            }
        ],
        "sample_group": {
            "date": "2015-08-24",
            "random_number": "55"
        },
        "another_group": {
            "another_repeat": [
                {
                    "sample_text": "Sample text 1",
                    "image": ""
                },
                {
                    "sample_text": "Sample text 2",
                    "image": ""
                }
            ]
        },
        "form_done": "OK",
        "survey_start": "2015-08-24T16:55:23.185+01",
        "survey_end": "2015-08-24T16:57:24.460+01",
        "survey_day": "2015-08-24",
        "survey_device": "353490061313389",
        "id": "ppe_checklist_new",
        "uuid": "2aba0eff-5350-47e3-9e9c-9606d2c9e7d6"
    }
}

I tried checking whether the data I am inserting to MongoDB is object or not and turns out that it is. 

typeof(result.result)
  1. Tried inserting the generated JSON object from xml2js via the Mongo Shell and it worked.
  2. Tried creating an JS object in my javascript code with the same content as JSON generated by xml2js and inserting that to MongoDB via mongodb native driver worked (even though I compared the two JSON and they were exactly the same).

In conclusion, I could say that mongodb native driver does not like the JSON object that is generated by xml2js module, why I do not know? I can't wait to know. I tried the following to work around

  1. Convert the XML 2 JSON using xml2js
  2. Stringify the returned JSON object from xml2js (x = JSON.stringify(obj);)
  3. Parsed the stringified JSON object back (parsedX = JSON.parse(x);)
  4. Inserted the parsed value into MongoDB via mongodb native drive, and it worked.

Regardless of what the issue, the above error message is kinda misleading or maybe not appropriate for this issue that is causing it. I could not find a way to get more detailed error (maybe there is a way that mongodb provide more explanation, would love to know).

Thank you for your time and patience.

1
jsonxmlnode.jsmongodbnode-mongodb-native
Well the very first line in the parsed object is basically { "ppe": { "$": { "id": "ppe_checklist_new" }, which is no doubt the source of the issue. Point is can you just not work from the very next element which is "starting_repreat": [ ? Or even just everything under { "ppe": { "$":? You also probably really don't want this as raw input into your database. It is a "database" afterall, and not an XML file. So there are other considerations here. – Blakes Seven
Try changing the o[i].$ = undefined; line to delete o[i].$;; – JohnnyHK
@JohnnyHK cheers, the delete did the trick. I am curious how JSON.stringify() followed by JSON.parse() approach worked. Does these methods do anything fancy to JSON object taking into account this issue? – Raf
Yes, the undefined value can't be represented in JSON so the JSON.stringify will remove any fields with that value. – JohnnyHK
@BlakesSeven if I use after ppe, then I will miss some of the fields (form_done, survey_start, etc). What do you mean by raw? I would definitely not insert the JSON as shown above to the database in which case the integer, date would also end up as strings. I will modify the JSON so all non-String fields are stored using MongoDB data types. Sorry, I am mean this MongoDB. Thank you for your comment. – Raf

1 Answers

2
votes

The following line of the traverse function just sets the $ field to a value of undefined without actually removing it.

o[i].$ = undefined;

Change that line to use delete to remove it instead:

delete o[i].$;

Your workaround using JSON.stringify and JSON.parse works because the undefined value can't be represented in JSON so the JSON.stringify call removes fields with that value.