Delete nested date folder getdate() < 5 date

问题

Folders in DataLake have nested date folder structure

I have wrote an Datafactory using actives for-each , GetMetaData,IfCondition and Delete

{
    "name": "IterateEachADLSItem",
    "type": "ForEach",
    "dependsOn": [
        {
            "activity": "F_SAP",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "userProperties": [],
    "typeProperties": {
        "items": {
            "value": "@activity('F_SAP').output.value",
            "type": "Expression"
        },
        "isSequential": false,
        "activities": [
            {
                "name": "GetPurgeFolderMetadata",
                "description": "GetPurgeFolderMetadata",
                "type": "GetMetadata",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "dataset": {
                        "referenceName": "DS_ADLS_FolderMetadata",
                        "type": "DatasetReference",
                        "parameters": {
                            "FolderPath": {
                                "value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-30),item().LoadIntervalFormat))",
                                "type": "Expression"
                            }
                        }
                    },
                    "fieldList": [
                        "exists"
                    ]
                }
            },
            {
                "name": "CheckPurgeFolderExists",
                "type": "IfCondition",
                "dependsOn": [
                    {
                        "activity": "GetPurgeFolderMetadata",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "expression": {
                        "value": "@activity('GetPurgeFolderMetadata').output.Exists",
                        "type": "Expression"
                    },
                    "ifTrueActivities": [
                        {
                            "name": "DeletePurgeFiles",
                            "type": "Delete",
                            "dependsOn": [],
                            "policy": {
                                "timeout": "7.00:00:00",
                                "retry": 0,
                                "retryIntervalInSeconds": 30,
                                "secureOutput": false,
                                "secureInput": false
                            },
                            "userProperties": [],
                            "typeProperties": {
                                "dataset": {
                                    "referenceName": "DS_ADLS_CopyDataSinkFile",
                                    "type": "DatasetReference",
                                    "parameters": {
                                        "SinkFilePath": {
                                            "value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-5),item().LoadIntervalFormat))",
                                            "type": "Expression"
                                        }
                                    }
                                },
                                "enableLogging": false,
                                "recursive": true
                            }
                        }
                    ]
                }
            }
        ]
    }
}

I am able to delete the exact 5 days before when it comes to date in the yyyy/mm/dd . if i execute on 2019/10/19, i am able to delete 2019/10/14

Delete is fine based on below syntax for exact 5 dates before

''"value": "@concat(item().DestinationPath,item().SinkTableName,'/',item().LoadTypeName,'/',formatDateTime(adddays(utcnow(),-5),item().LoadIntervalFormat))",

I am not able to loop and find for earlier days in the folder for each month or earlier year provided there is an folder existing.

ie : how to delete for previous month and previous years when we are in say eg: 03/01/2020.

回答1:

I think you could accomplish your goal by below approach:

Have Get Metadata activity to list all folders within root folder, and include foreach activity within GetMetadata.This should produce list of years.
Include Condition activity to compare the current year against folder's name and include delete activity,if that number is less than the current year.
If the above condition doesn't match, include another condition to check if it's equal and if yes, include another Get Metadata activity,
The above Get Metadata activity will list folders, which should produce months list and include foreach activity. Have similar logic for the months and include another metadata activity to list days for the current month.
1. Finally, the last Get Metadata activity will list days and when day number is less than current day, will include Delete activity

Hope that helps, Fikrat

来源：https://stackoverflow.com/questions/58462309/delete-nested-date-folder-getdate-5-date

标签

azure

azure-data-factory