Pymongo get inserted id's even with duplicate key error

Question

I am working on a flask app and using mongodb with it. In one endpoint i took csv files and inserts the content to mongodb with insert_many() . Before inserting i am creating a unique index for preventing duplication on mongodb. When there is no duplication i can reach inserted_ids for that process but when it raises duplication error i get None and i can't get inserted_ids . I am using ordered=False also. Is there any way that allows me to get inserted_ids even with duplicate key error ?

def createBulk(): #in controller
  identity = get_jwt_identity()
  try:
    csv_file = request.files['csv']
    insertedResult = ProductService(identity).create_product_bulk(csv_file)
    print(insertedResult) # this result is None when get Duplicate Key Error
    threading.Thread(target=ProductService(identity).sendInsertedItemsToEventCollector,args=(insertedResult,)).start()
    return json_response(True,status=200)
  except Exception as e:  
    print("insertedResultErr -> ",str(e))
    return json_response({'error':str(e)},400)

def create_product_bulk(self,products): # in service
        data_frame = read_csv(products)
        data_json = data_frame.to_json(orient="records",force_ascii=False)
        try:
            return self.repo_client.create_bulk(loads(data_json))
        except bulkErr as e:
            print(str(e))
            pass
        except DuplicateKeyError as e:
            print(str(e))
            pass

def create_bulk(self, products): # in repo
        self.checkCollectionName()
        self.db.get_collection(name=self.collection_name).create_index('barcode',unique=True)
        return self.db.get_collection(name=self.collection_name).insert_many(products,ordered=False)

Belly Buster Belly Buster · Accepted Answer · 2020-08-05T11:54:40

Unfortunately, not in the way you have done it with the current pymongo drivers. As you have found, if you get errors in your insert_many() it will throw an exception and the exception detail does not contain details of the inserted_ids.

It does contain details of the keys the fail (in e.details['writeErrors'][]['keyValue']) so you could try and work backwards from that from your original products list.

Your other workaround is to use insert_one() in a loop with a try ... except and check each insert. I know this is less efficient but it's a workaround ...

Pymongo get inserted id's even with duplicate key error

1 Answers