Azure MongoDB API Bulk Insert

Question

I currently make use of Azure Mongo API for documentDB. However this has been causing us many headaches such as inserting documents into the db and trying to ignore the duplicate key errors during an InsertMany(). If there is a document within the batch that is a duplicate it immediately throws and error and returns even though I would require it to carry on. I am aware why we are getting these as I am busy moving data across from a non partitioned collection to a partitioned collection and there is a good chance they already have been migrated.

Apparently during the insert many you can pass isOrdered = false within the insertmanyoptions() and it should do this, it did not.

After eventually giving up, I attempted to try this via a BulkWriteAsync whilst making use of the IsUpsert = true Sadly I get a funny error:

Value cannot be null. Parameter name : requests

Here is my code below.

    public async Task BulkInsertDeviceConfigData(List<DeviceEventConfigure> Elements)
    {
        var models = new List<WriteModel<BsonDocument>>();

        // use ReplaceOneModel with property IsUpsert set to true to upsert whole documents
        foreach (var element in Elements)
        {
            var bsonDoc = element.ToBsonDocument();
            models.Add(new ReplaceOneModel<BsonDocument>(new BsonDocument("_id", element.Id), bsonDoc) { IsUpsert = true });
        };

        await BulkWriteAsync(models as IEnumerable<WriteModel<DeviceEventData>>);
    }

    public virtual async Task BulkWriteAsync(IEnumerable<WriteModel<T>> models)
    {
        await Collection.BulkWriteAsync(models);
    }

This has been hurting me as we have been increasing our RUs a lot lately to do this even though sadly have been unsuccessful.

Is this an Azure Mongo API issue or is it me?

It is commonplace for code and console/io to be provided as text rather than images, so I'd suggest changing the first image here. Images are not compatible with clipboards, search engines and screen-readers, and so are not as easy for readers to work with. — halfer
Lol how much money has your company wasted on documentDB so far? The pricing for it is ridiculous for something which is literally a mongodb clone. Just rent a few VMs and set up a mongo cluster. Cheaper and not charged by the collection (!!!!). Can you not filter out duplicates? Why do you have duplicates in the first place? What is your contingency for duplication? Does first, or last, win? — Mardoxx
@Mardoxx haha yeah it is rather expensive but great performance benefits. Scaling is great too! Anyway Im now considering moving my db context from mongo to docdb as just not winning with this API. Had endless issues — David

Martin Dekker Martin Dekker · Accepted Answer · 2020-03-05T15:00:35

Maybe your cast

models as IEnumerable<WriteModel<DeviceEventData>>

isn't working resulting in a null value for the parameter requests. I don't think casting to a BsonDocument is necessary, so it can be written as:

public async Task BulkInsertDeviceConfigData(List<DeviceEventConfigure> Elements)
{
    var models = new List<WriteModel<DeviceEventConfigure>>();

    // use ReplaceOneModel with property IsUpsert set to true to upsert whole documents
    foreach (var element in Elements)
    {
        models.Add(new ReplaceOneModel<DeviceEventConfigure>(new BsonDocument("_id", element.Id), element) { IsUpsert = true });
    };

    await BulkWriteAsync(models as IEnumerable<WriteModel<DeviceEventConfigure>>);
}

public virtual async Task BulkWriteAsync(IEnumerable<WriteModel<T>> models)
{
    await Collection.BulkWriteAsync(models);
}

Azure MongoDB API Bulk Insert

1 Answers