Is there a way to insert a document with a nested array in Azure Data Factory?

问题

I am trying to add documents in CosmosDb that has a nested array. I am using the Copy Activity.

Sample Document:

{
  "itemNumber": "D10001",      
  "readings" : [
                  { "value": 25, "ets":"100011111"},
                  { "value": 35, "ets":"100011122"}
               ]
}

In the source dataset I formatted the readings array as a string in my SQL query, and set the data type in the sink dataset as an Object. The data is copied, but the readings are stringified.

Is there a means to configure the Copy Activity to handle this array?

回答1:

What is your source? You could copy your data to json files first. And then import it to cosmos DB as is, which means don’t specify the format and structure in your source and sink dataset.

回答2:

As I know, no such properties could help you convert string data to array format in adf cosmos db configuration.

Since you are using adf to import data so that you can't use PreTrigger to change the format of created documents.PreTrigger need to be invoked by code or rest api.

So, as workaround, I suggest you using Azure Function Cosmos DB Trigger to process every document when they imported into database. Please refer to my function code:

using System.Collections.Generic;
using Microsoft.Azure.Documents;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json.Linq;
using System;
using Microsoft.Azure.Documents.Client;

namespace TestADF
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([CosmosDBTrigger(
            databaseName: "db",
            collectionName: "item",
            ConnectionStringSetting = "documentdbstring",
            LeaseCollectionName = "leases")]IReadOnlyList<Document> input, TraceWriter log)
        {
            if (input != null && input.Count > 0)
            {
                log.Verbose("Start.........");
                String endpointUrl = "https://***.documents.azure.com:443/";
                String authorizationKey = "key";
                String databaseId = "db";
                String collectionId = "item";

                DocumentClient client = new DocumentClient(new Uri(endpointUrl), authorizationKey);

                for (int i = 0; i < input.Count; i++)
                {
                    Document doc = input[i];
                    if ((doc.GetPropertyValue<Boolean>("alreadyForamt") == null) || (!doc.GetPropertyValue<Boolean>("alreadyForamt")))
                    {                       
                        String readings = doc.GetPropertyValue<String>("readings");
                        JArray r = JArray.Parse(readings);

                        doc.SetPropertyValue("readings", r);

                        client.ReplaceDocumentAsync(UriFactory.CreateDocumentUri(databaseId, collectionId, doc.Id), doc);

                        log.Verbose("Update document Id " + doc.Id);
                    }

                }
            }
        }
    }
}

Hope it helps you.

来源：https://stackoverflow.com/questions/51430699/is-there-a-way-to-insert-a-document-with-a-nested-array-in-azure-data-factory

标签

azure-cosmosdb

azure-data-factory