python - Remove duplicates from json data

Question

Is there an efficient way to remove duplicates 'person_id' fields from this data with python? In this case just keep the first occurrence.

{
  {obj_id: 123,
    location: {
      x: 123,
      y: 323,
  },
  {obj_id: 13,
    location: {
      x: 23,
      y: 333,
  },
 {obj_id: 123,
    location: {
      x: 122,
      y: 133,
  },
}

Should become:

{
  {obj_id: 123,
    location: {
      x: 123,
      y: 323,
  },
  {obj_id: 13,
    location: {
      x: 23,
      y: 333,
  },
}

Answer 1

Presuming your JSON is valid syntax and you are indeed requesting help for Python you will need to do something like this

import json
ds = json.loads(json_data_string) #this contains the json
unique_stuff = { each['obj_id'] : each for each in ds }.values()

if you want to always retain the first occurrence, you will need to do something like this

all_ids = [ each['obj_id'] for each in ds ] # get 'ds' from above snippet
unique_stuff = [ ds[ all_ids.index(id) ] for id in set(ids) ]

Answer 2

Here's an implementation that preserves order of input json objects and keeps the first occurrence of objects with the same id:

import json
import sys
from collections import OrderedDict

L = json.load(sys.stdin, object_pairs_hook=OrderedDict)
seen = OrderedDict()
for d in L:
    oid = d["obj_id"]
    if oid not in seen:
        seen[oid] = d

json.dump(seen.values(), sys.stdout,  indent=2)

Input

[
  {
    "obj_id": 123, 
    "location": {
      "x": 123, 
      "y": 323
    }
  }, 
  {
    "obj_id": 13, 
    "location": {
      "x": 23, 
      "y": 333
    }
  }, 
  {
    "obj_id": 123, 
    "location": {
      "x": 122, 
      "y": 133
    }
  }
]

Output

[
  {
    "obj_id": 123, 
    "location": {
      "x": 123, 
      "y": 323
    }
  }, 
  {
    "obj_id": 13, 
    "location": {
      "x": 23, 
      "y": 333
    }
  }
]

Answer 3

(if you had valid json)

from simplejson import loads, dumps
dumps(loads(my_json))

Answer 4

This is not valid JSON. On a valid JSON (Array), You can use jQuery $.each and look at the Obj_id to find and remove duplicates.

Something like this:

$.each(myArrayOfObjects, function(i, v)
{
      // check for duplicate and add non-repeatings to a new array
});