azure-cosmosdb

Azure Data Factory Copy activity/Data flow consumes all RUs in CosmosDB

心已入冬 提交于 2021-02-19 05:49:12
问题 We are using Azure Data Factory for ETL to push materialized views to our Cosmos DB instance, making sure our production Azure CosmosDB (SQL API) is containing all necessary for our users. The CosmosDB is under constant load as data flows in via speed layer as well. This is expected and currently solved with an autoscaling RU setting. We have tried these options: An ADF pipeline with Copy activity to upsert data from Azure Datalake (Gen2) (source) to the collection in Cosmos DB (sink). An ADF

How do I get the latest record for each item in CosmosDB using SQL

微笑、不失礼 提交于 2021-02-16 15:36:29
问题 I have a schema which is similar to "id": "uuid", "deviceId": "uuid", "message": { "content": "string", "ts": 1 }, "data": { "temperature": 21 } I'd like to get the latest "data" (using message.ts as the timestamp) for each "deviceId". So far, I've managed to get the data back, in order of timestamp using the query SELECT c.deviceId, c.message.ts, c.data FROM c ORDER BY c.message.ts DESC but I can't figure out how to remove the duplicate device records. Is this possible to do within the

EF Core with Cosmos DB provider, UserManager AddLoginAsync gives ConcurrencyFailure

亡梦爱人 提交于 2021-02-15 07:35:24
问题 Creating a new user, fetching the user again with usermanager for testing, and then using the method AddLoginAsync with the recently fetched user gives the error ConcurrencyFailure, Optimistic concurrency failure, object has been modified. When fetching the user the "ConcurrencyStamp" has the correct etag, but after the "AddLoginAsync" I can see the user object has an invalid etag, the ConcurrencyStamp is a GUID. I have followed the documentation and added this to the IdentityUser model

Can we have multiple readers(reading records through pagination) in a single batch job

巧了我就是萌 提交于 2021-02-11 17:52:25
问题 I have a spring batch job to be written . where : I need to read say 10k records through pagination(fetching 1000 rec at a time) (from azure SQL db) I need to use 1000 records at a time and then use one column of these records(say some id) to read corresponsing records from another cosmos db table. How do I implement 2 readers in this case as I need to read 1000 records at a time and again fetch records from cosmos db for those 1000 records first and process that. 回答1: There is a common

How To Update Record in Cosmos Db using python?

只愿长相守 提交于 2021-02-11 16:42:31
问题 I want to read record from cosmos db and update same record in cosmos db using python. example: { "id":"id-1", "name":"mohit" } I want to read the above record and update it to. { "id":"id-1", "name":"Mohit Singh" } I find few link but seems they only create or delete the record but not updating the existing record: https://github.com/Azure/azure-cosmos-python#modify-container-properties but unable to get how to update existing record. in my scenario i want to read all the record from cosmos

How To Update Record in Cosmos Db using python?

元气小坏坏 提交于 2021-02-11 16:42:19
问题 I want to read record from cosmos db and update same record in cosmos db using python. example: { "id":"id-1", "name":"mohit" } I want to read the above record and update it to. { "id":"id-1", "name":"Mohit Singh" } I find few link but seems they only create or delete the record but not updating the existing record: https://github.com/Azure/azure-cosmos-python#modify-container-properties but unable to get how to update existing record. in my scenario i want to read all the record from cosmos

Query last data of different documents in Azure Cosmos db (SQL)

这一生的挚爱 提交于 2021-02-11 14:38:38
问题 Sample data : {age: 20, ts: '00:00'}, {age: 20, ts: '00:01'}, {age: 30, ts: '00:00'}, {age: 30, ts: '00:01'}, {age: 40, ts: '00:00'}, {age: 40, ts: '00:01'}, {age: 40, ts: '00:02'} Expected output: [{age: 20, ts: '00:01'}, {age: 30, ts: '00:01'}, {age: 40, ts: '00:02'}] Tried SELECT * FROM c where c.age in (20, 30, 40) order by c.ts desc But the result selected all data. 回答1: Please try something like the following: SELECT max(c.ts) as ts, c.age FROM c where c.age in (20, 30, 40) Group By c

EF Core with Cosmos DB provider, UserManager AddLoginAsync gives ConcurrencyFailure

岁酱吖の 提交于 2021-02-11 12:49:46
问题 Creating a new user, fetching the user again with usermanager for testing, and then using the method AddLoginAsync with the recently fetched user gives the error ConcurrencyFailure, Optimistic concurrency failure, object has been modified. When fetching the user the "ConcurrencyStamp" has the correct etag, but after the "AddLoginAsync" I can see the user object has an invalid etag, the ConcurrencyStamp is a GUID. I have followed the documentation and added this to the IdentityUser model

Migrate JSON data from Azure SQL DB to Cosmos DB results in string values

匆匆过客 提交于 2021-02-11 12:20:42
问题 I'm trying to migrate data from SQL DB using CosmosDB Data Migration Tool and I successfully migrated data from SQL DB but the result is all values are string Wondering if there's a way to convert those JSON to Object during migration process? Here's my sample Query select json_value(Data, '$.timestamp') as timestamp, json_query(Data, '$.Product.detail') as [Product.detail], json_value(Data, '$.Product.price') as [Product.price] from myTable nesting seperator: . 回答1: 1.create a dataflow and

Deserialize JSON into multiple inherited classes

孤人 提交于 2021-02-11 09:22:49
问题 When I serialize my JSON object out of DocumentDB, my Control is not deserializing into the OptionsControl with the Options property. I have the following class, Control : public class Control : IControl { public Guid Id { get; set; } public virtual Enums.ControlType Type { get; set; } public string PropertyName { get; set; } public string ControlCssClass { get; set; } public string Description { get; set; } } I also have OptionsControl , which inherits from Control : public class