Want to capture Data Load Log to a SQL or Snowflake table in Mule Flow

两盒软妹~` 提交于 2020-01-30 10:57:27

问题


<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
	xmlns:db="http://www.mulesoft.org/schema/mule/db"
	xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd">
	<flow name="From_SQL_Table_To_Snowflake_Table_FullLoad_Flow" doc:id="d1a52bb0-1014-4d18-8528-7077b9807c7f" >
		<scheduler doc:name="Scheduler - Runs Every 10 Seconds" doc:id="5ac8b208-57ec-4b58-aa7c-c411bde3c1ce" >
			<scheduling-strategy >
				<fixed-frequency frequency="10" timeUnit="SECONDS"/>
			</scheduling-strategy>
		</scheduler>
		<db:select doc:name="Select Flow Configuration" doc:id="cd8588b6-0458-430e-9779-3bba33f02065" config-ref="Database_Config-1" queryTimeout="10" queryTimeoutUnit="MINUTES">
			<db:sql >SELECT [Id], 
       [GroupId], 
       [GroupName], 
       [SourceTable], 
       [SourceQuery], 
       [DestinationTable], 
       [DestinationQuery], 
       [IsIncluded], 
       [IsManualRun], 
       [FullLoadFlag], 
       [IsTruncate], 
       [TruncateQuery], 
       [DeltaLoadFlag], 
       [IsDelete], 
       [DeleteQuery], 
       [DeltaColumn], 
       [DeltaValueFrom], 
       [DeltaValueTo], 
       [Comments], 
       [CreatedDateTime], 
       [ModifiedDateTime]
FROM [dbo].[Shivendoo_Config]
WHERE [GroupName] = 'Full Load-1'
      AND [IsIncluded] = 1
	  AND [FullLoadFlag] = 1
	  AND [IsTruncate] = 1</db:sql>
		</db:select>
		<logger level="INFO" doc:name="Logger to Get Size Of Payload" doc:id="60cbd4bc-f077-4854-9156-d5726561eb0e" message="Size Of Payload: #[sizeOf(payload)]" />
		<batch:job jobName="Master_Flow_To_Load_Data_From_SQL_Tables_To_Snowflake_TablesBatch_Job" doc:id="7381a2dd-d194-478a-bb22-31c244a3fedb" >
			<batch:process-records >
				<batch:step name="Batch_Step" doc:id="25105496-4d87-48f6-b86b-ec8635ff01fe" >
					<set-variable value="#[payload.SourceQuery]" doc:name="Set Variable to Get SourceQuery" doc:id="87ff6d56-663b-4783-a76c-a9c7f88857e4" variableName="VSourceQuery"/>
					<set-variable value="#[payload.DestinationQuery]" doc:name="Set Variable to Get DestinationQuery" doc:id="f335c1f3-9a2b-4544-8ad7-7ccb11a20977" variableName="VDestinationQuery"/>
					<set-variable value="#[payload.TruncateQuery]" doc:name="Set Variable to Get TruncateQuery" doc:id="9b359b5e-8280-4e26-88ee-8e7b68215782" variableName="VTruncateQuery"/>
					<logger level="INFO" doc:name="Logger to Get SourceQuery and DestinationQuery From Batch" doc:id="13ceec10-acc6-4308-b135-833b21c34e40" message="Source Table and DestinationQuery From Batch: #[vars.VSourceQuery] #[vars.VDestinationQuery]" />
					<db:execute-script doc:name="Execute script To Truncate Destination Table" doc:id="4fa31672-f5b6-4432-a490-128950c970f8" config-ref="Database_Config-1">
						<db:sql >#['$(vars.VTruncateQuery)'] </db:sql>
					</db:execute-script>
					<db:select doc:name="Select From Table Dynamic" doc:id="187ac9b7-9b94-4c97-b893-7113ab903104" config-ref="Database_Config-1">
				<db:sql>#['$(vars.VSourceQuery)'] </db:sql>
			
</db:select>
					<logger level="INFO" doc:name="Logger to Get Size Of Payload" doc:id="a7b61a28-192d-4c7c-bad2-dcf04f791b00" message="Record Counts: #[payload.successfulRecords]" />
					<db:bulk-insert doc:name="Bulk insert to Dynamic Tables" doc:id="7a7419d8-f1df-41b0-b705-7fc9a6f92fce" config-ref="Database_Config-1">
				<db:sql>#[' $(vars.VDestinationQuery)']
</db:sql>
			</db:bulk-insert>
				</batch:step>
			</batch:process-records>
		</batch:job>
	</flow>

</mule>

I have a mule flow Which loads data from Source to Destination SQL Table.

In between Source to Destination I want to capture No of Records inserted into Destination. Below is my Table Structure. I need following information: Flow_Name Insert_Rowcnt OR Update_Rowcnt OR Delete_Rowcnt

I already have following information into Variables: Source_Table Destination_Table Source_Query Destination_Query Start_DT Finish_DT

Table Structure:

This is My Flow

来源:https://stackoverflow.com/questions/59870415/want-to-capture-data-load-log-to-a-sql-or-snowflake-table-in-mule-flow

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!