CosmosDb Request rate is large with insertMany

问题

I have the following repository class inserting data in a CosmosDb database from a batch :

public bool InsertZonierData(List<Zonier> zonierList)
{
    if (zonierList == null || !zonierList.Any())
    {
        throw new ZonierListNullOrEmptyException();
    }
    else
    {
        try
        {
            _collection.InsertMany(zonierList);
            return true;
        }
        catch (MongoBulkWriteException ex)
        {
            throw new DataBaseWritingException(ex.Message, ExceptionCodeConstants.DataBaseWritingExceptionCode);
        }
    }
}

Unfortunately, having more than 30000 elements in the zonierList, it throw the following exception on CosmosDb :

Unhandled Exception: MongoDB.Driver.MongoCommandException: Command insert failed: Message: {"Errors":["Request rate is large"]}

Following the documentation, it is an issue related to RU / sec on Cosmos. Of course an easy way would be to increase it, but that's not what I want to do.

Is there an easy and clear way to refactor the method allowing us to insert data without breaking the 400 RU / sec from CosmosDb.

回答1:

The mongo sdk is completely unaware of the CosmosDB’s existence. That means that it doesn’t have any retry logic for throttled requests. That means that, if you wanna keep the RUs to 400, you would have to batch your list and call the insertmany method using a client side throttling mechanism.

You can calculate that by getting the size of each document, multiply that with 10 which is the insert charge for 1kb of document and then write a piece of code that batches documents based on the size and executes once per second.

回答2:

Mongo driver tells you which records got errors and which were not processed at all. If all errors (usually one) have code 16500, then your problem is throttling and retry on errors and remaining records is safe. Otherwise your errors are caused by something else and you should do analysis and decide whether to continue with retries.

Mongo driver does not return HTTP header where Cosmos DB suggests delay before retry, but that is not a big deal. Delay does not guarantee success anyway, because other requests hitting same database may use up RUs. You are better off experimenting and determining your own retry rules. Below is simple recursive solution that keeps retrying till all is well or retry limit is reached.

    private async Task InsertManyWithRetry(IMongoCollection<BsonDocument> collection, 
        IEnumerable<BsonDocument> batch, int retries = 10, int delay = 300)
    {
        var batchArray = batch.ToArray();

        try
        {
            await collection.InsertManyAsync(batchArray);
        }
        catch (MongoBulkWriteException<BsonDocument> e)
        {
            if (retries <= 0)
                throw;

            //Check if there were any errors other than throttling.
            var realErrors = e.WriteErrors.Where(we => we.Code != 16500).ToArray();
            //Re-throw original exception for now.
            //TODO: We can make it more sophisticated by continuing with unprocessed records and collecting all errors from all retries.
            if (realErrors.Any())
                throw;

            //Take all records that had errors.
            var errors = e.WriteErrors.Select(we => batchArray[we.Index]);
            //Take all unprocessed records.
            var unprocessed = e.UnprocessedRequests
                .Where(ur => ur.ModelType == WriteModelType.InsertOne)
                .OfType<InsertOneModel<BsonDocument>>() 
                .Select(ur => ur.Document);

            var retryBatchArray = errors.Union(unprocessed).ToArray();

            _logger($"Retry {retryBatchArray.Length} records after {delay} ms");

            await Task.Delay(delay);

            await InsertManyWithRetry(collection, retryBatchArray, retries - 1, delay);
        }
    }

回答3:

I solved it by using retry logic for cosmos bs using mongo api. You can apply delay as per your requirements.

public void Insert(List<BsonDocument> list)
    {
        try
        {
            var collection = this.db.GetCollection<BsonDocument>(COLLECTION_NAME);
            collection.InsertMany(list);
        } catch (MongoBulkWriteException ex)
        {
            int index = ex.WriteErrors[0].Index;
            Insert(list.GetRange(index, list.Count - index));
        }

    }

来源：https://stackoverflow.com/questions/55263897/cosmosdb-request-rate-is-large-with-insertmany