SQL Server complains about invalid json

问题

I am writing an ETL tool using Azure Data Factory and Azure SQL Database. The Data Factory captures the output of a Mapping Data Flow and inserts it into the StatusMessage column of a SQL Server table (Audit.OperationsEventLog) as a string. The StatusMessage column is varchar(8000) and is intended to store data formatted as valid json.

SELECT *
FROM Audit.OperationsEventLog lg
CROSS APPLY OPENJSON(lg.StatusMessage) dt

When I query the json string from the table using the query above, it complains

JSON text is not properly formatted. Unexpected character '"' is found at position 382

That's a double-quote surrounded by two single-quotes.

I have used JSONLint (http://jsonlint.com) to validate that the json string is encoded properly.

When I copy the json string from the column StatusMessage into a varchar(8000) variable, I'm able to parse the string using OPENJSON.

DECLARE @testjson varchar(8000) = '
{   "EventType": "DataFactoryPipelineRunActivity",    
    "DataFactoryName":"fa603ea7-f1bd-48c0-a690-73b92d12176c",   
    "DataFactoryPipelineName":"Import Blob Storage Account Key CSV file into generic SQL table using Data Flow Activity Logging to Target SQL Server",   
    "DataFactoryPipelineActivityName":"Copy Generic CSV Source to Generic SQL Sink",   
    "DataFactoryPipelineActivityOutput":"{runStatus:{computeAcquisitionDuration:316446,dsl: source() ~> ReadFromCSVInBlobStorage  ReadFromCSVInBlobStorage derive() ~> EnrichWithDataFactoryMetadata  EnrichWithDataFactoryMetadata sink() ~> WriteToTargetSqlTable,profile:{ReadFromCSVInBlobStorage:{computed:[],lineage:{},dropped:0,drifted:1,newer:1,total:1,updated:0},EnrichWithDataFactoryMetadata:{computed:[],lineage:{},dropped:0,drifted:1,newer:6,total:7,updated:0},WriteToTargetSqlTable:{computed:[],lineage:{__DataFactoryPipelineName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryPipelineName]}]},__DataFactoryPipelineRunId:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryPipelineRunId]}]},id:{mapped:true,from:[{source:ReadFromCSVInBlobStorage,columns:[id]}]},__InsertDateTimeUTC:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__InsertDateTimeUTC]}]},__DataFactoryName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__DataFactoryName]}]},__FileName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__FileName]}]},__StorageAccountName:{mapped:false,from:[{source:EnrichWithDataFactoryMetadata,columns:[__StorageAccountName]}]}},dropped:0,drifted:1,newer:0,total:7,updated:7}},metrics:{WriteToTargetSqlTable:{rowsWritten:4,sinkProcessingTime:1436,sources:{ReadFromCSVInBlobStorage:{rowsRead:4}},stages:[{stage:3,partitionTimes:[621],bytesWritten:0,bytesRead:24,streams:{WriteToTargetSqlTable:{type:sink,count:4,partitionCounts:[4],cached:false},EnrichWithDataFactoryMetadata:{type:derive,count:4,partitionCounts:[4],cached:false},ReadFromCSVInBlobStorage:{type:source,count:4,partitionCounts:[4],cached:false}},target:WriteToTargetSqlTable,time:811}]}}},effectiveIntegrationRuntime:DefaultIntegrationRuntime (East US)}",   
    "DataFactoryPipelineRunID":"63759585-4acb-48af-8536-ae953efdbbb0",   
    "DataFactoryPipelineTriggerName":"Manual",   
    "DataFactoryPipelineTriggerType":"Manual",   
    "DataFactoryPipelineTriggerTime":"2019-11-05T15:27:44.1568581Z",   
    "Parameters":{    
        "StorageAccountName":"fa603ea7",     
        "FileName":"0030_SourceData1.csv",    
        "TargetSQLServerName":"5a128a64-659d-4481-9440-4f377e30358c.database.windows.net",     
        "TargetSQLDatabaseName":"TargetDatabase",     
        "TargetSQLUsername":"demoadmin"   
    },    
    "InterimValues":{    
        "SchemaName":"utils",     
        "TableName":"vw_0030_SourceData1.csv-2019-11-05T15:27:57.643"   
    }  
}'

SELECT      *
FROM        OPENJSON(@testjson)

SELECT      *
FROM        OPENJSON(@testjson) data
            CROSS APPLY OPENJSON(data.value) moredata
WHERE       data.type = 5

The problem is isolated to "DataFactoryPipelineActivityOutput".

The data factory builds the json string to be inserted into the StatusMessage column of the table. I strip out any occurrences of double-quotes within the StatusMessage string.

{
    "EventDateTime":"@{utcNow()}",
    "EventState":"Success",
    "SourceName":"@{concat(pipeline().DataFactory, '/', pipeline().Pipeline, '/Copy Generic CSV Source to Generic SQL Sink')}",
    "SourceType":"DataFactoryPipelineRunActivity",
    "StatusMessage":"{
        \"EventType\": \"DataFactoryPipelineRunActivity\", 
        \"DataFactoryName\":\"@{pipeline().DataFactory}\",
        \"DataFactoryPipelineName\":\"@{pipeline().Pipeline}\",
        \"DataFactoryPipelineActivityName\":\"Copy Generic CSV Source to Generic SQL Sink\",
        \"DataFactoryPipelineActivityOutput\":\"@{replace(string(activity('Copy Generic CSV Source to Generic SQL Sink').output), '"', '')}\",      \"DataFactoryPipelineRunID\":\"@{pipeline().RunID}\",
        \"DataFactoryPipelineTriggerName\":\"@{pipeline().TriggerName}\",
        \"DataFactoryPipelineTriggerType\":\"@{pipeline().TriggerType}\",
        \"DataFactoryPipelineTriggerTime\":\"@{pipeline().TriggerTime}\",
        \"Parameters\":{
            \"StorageAccountName\":\"@{pipeline().parameters.StorageAccountName}\", 
            \"FileName\":\"@{pipeline().parameters.FileName}\",
            \"TargetSQLServerName\":\"@{pipeline().parameters.TargetSQLServerName}\", 
            \"TargetSQLDatabaseName\":\"@{pipeline().parameters.TargetSQLDatabaseName}\", 
            \"TargetSQLUsername\":\"@{pipeline().parameters.TargetSQLUsername}\"
        }, 
        \"InterimValues\":{
            \"SchemaName\":\"@{activity('Get Target View Schema and Name').output.firstRow.SchemaName}\", 
            \"TableName\":\"@{activity('Get Target View Schema and Name').output.firstRow.ViewName}\"
        }
    }"
}

Can anyone see if I'm doing something wrong, or is this a bug in OPENJSON? I'm hoping that I did something stupid and all I need is a second set of eyes

回答1:

If the sequence

 '"'

is part of a string, shouldn't that be

'\"'

otherwise it's interpreted as the end of string, and that would indeed be invalid JSON.

So, that means

\"DataFactoryPipelineActivityOutput\":\"@{replace(string(activity('Copy Generic CSV Source to Generic SQL Sink').output), '\"', '')}\",      \"DataFactoryPipelineRunID\":\"@{pipeline().RunID}\",

instead of

\"DataFactoryPipelineActivityOutput\":\"@{replace(string(activity('Copy Generic CSV Source to Generic SQL Sink').output), '"', '')}\",      \"DataFactoryPipelineRunID\":\"@{pipeline().RunID}\",

来源：https://stackoverflow.com/questions/58716310/sql-server-complains-about-invalid-json