azure-data-lake

问题 I am working on a backup strategy for Data Lake Store (DLS). My plan is to create two DLS accounts and copy data between them. I have evaluated several approaches to achieve this but none of them satisfies the requirement to preserve the POSIX ACLs (permissions in DLS parlance). PowerShell cmdlets require data to be downloaded from the primary DLS onto a VM and re-uploaded onto the secondary DLS. The AdlCopy tool works only on Windows 10, does not preserve permissions and neither supports

问题 Can we append data in existing file in U-SQL? I have created a CSV file as output in U-SQL. I am writing another U-SQL query and I want to append the output of that query in the existing file. Is it possible? 回答1: It's not supported, and would go against the design of a robust, distributed, idempotent big data system (although you could implement that behaviour by reading the previous output as a rowset and do UNION ALL). The best way to deal with this is to use partitions properly, for

问题 Can we append data in existing file in U-SQL? I have created a CSV file as output in U-SQL. I am writing another U-SQL query and I want to append the output of that query in the existing file. Is it possible? 回答1: It's not supported, and would go against the design of a robust, distributed, idempotent big data system (although you could implement that behaviour by reading the previous output as a rowset and do UNION ALL). The best way to deal with this is to use partitions properly, for

问题 I have an Azure Data Lake Storage (Gen 2) account with several containers. I would like to import the salesorderdetail.csv file from the Sales container into an Azure SQL database. I've successfully built the same process using Azure Data Factory, but I now want to try and get this working via standard T-SQL statements only. CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'XxxxxxXX#' CREATE DATABASE SCOPED CREDENTIAL MK_Cred_Data_Load WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = 'sv

问题 I am trying to read the content from the Azure Data Lake Store file. But while connecting / opening the connection itself fails & exception is thrown var stream = _adlsFileSystemClient.FileSystem.Open(_adlsAccountName, "/folder1/"+file.PathSuffix); Getting exception as: Exception of type 'Microsoft.Rest.Azure.CloudException' was thrown. Unable to recognize the issue. How to resolve this issue? Thank you 回答1: Exception of type 'Microsoft.Rest.Azure.CloudException' was thrown. It is basic

问题 I have created ADLS (Azure Data Lake Storage) Gen2 resource (StorageV2 with hierarchical name space enabled). The region I created the resource in is Central US and the performance/access tier is Standard/Hot and replication is LRS. But for this resource I can't see 'Lifecycle management' option on the portal. ADLS Gen2 is simply a StorageV2 account with hierarchical namespace enabled, and since the lifecycle management option exists for StorageV2 as per Microsoft documentation , it should be

问题 We can extend U-SQL scripts with R/Python code in Azure Data Lake Analytics, but how can we do it locally? 回答1: Install U-SQL Advanced Analytics extensions in your Data Lake Analytics Account 1.1 Launch your Azure Portal 1.2 Navigate to your Data Lake Analytics Account 1.3 Click Sample Scripts 1.4 Click More and select Install U-SQL Extensions 1.5 Wait until the extensions have finished installing (2GB) 1.6 Have you waited? Then go to your Data Lake Analytics Account 1.7 Navigate to your

问题 I am using flink to read the data from Azure data lake.But flink is not able to find the Azure data lake file system. how to configure flink to understand the Azure Data lake file system.Could anyone guide me in this? 回答1: Flink has the capability to connect to any Hadoop compatible file system (i.e that implements org.apache.hadoop.fs.FileSystem). See here for the explanation: https://ci.apache.org/projects/flink/flink-docs-release-0.8/example_connectors.html In the core-site.xml, you should

问题 I am trying to run a one-way hash for a string column in USQL. Is there a way to do this inline? Most of the C# samples found online require multiple lines of code - which is tricky in USQL without a code-behind or compiled C# assembly. 回答1: Option 1 (Inline formula): The code below can be used to compile a SHA256 or MD5 on any string, and runs without any special dependencies and without needing a code-behind file. CREATE TABLE master.dbo.Test_MyEmail_Hashes AS SELECT cust.CustEmailAddr AS

问题 I am trying to access a Data Lake from a .Net client application using this example I have registered the Client App in AAD Tenant and using the Client Id and Client secret from there (as what i believe is service-to-service authentication.) The Data Lake is in a different subscription but belongs to the same Tenant/AAD The App has Read/Write/Execute permission under 'Owner' and 'Assigned Permissions' for the specific folder (two hierarchies down the root folder) in the datalake. The parent