azure-data-lake | 易学教程

Transfer data from U-SQL managed table to Azure SQL Database table

阅读更多关于 Transfer data from U-SQL managed table to Azure SQL Database table

问题 I have a U-SQL managed table that contains schematized structured data. CREATE TABLE [AdlaDb].[dbo].[User] ( UserGuid Guid, Postcode string, Age int? DateOfBirth DateTime?, ) And a Azure SQL Database table. CREATE TABLE [SqlDb].[dbo].[User] ( UserGuid uniqueidentifier NOT NULL, Postcode varchar(15) NULL, Age int NULL, DateOfBirth Date NULL, ) I would like to transfer data from U-SQL managed table to Azure SQLDB table without losing the data types. I'm using azure data factory, seems like I

USQL - How To Select All Rows Between Two String Rows in USQL

阅读更多关于 USQL - How To Select All Rows Between Two String Rows in USQL

问题 Here is my complete task description: I have to extract data from multiple files using u-sql and output it into csv file. Every input file contains multiple reports based on some string rows ("START OF ..." and "END OF ..." working as report separator). Here is an example (data format) of a single source (input) file : START OF DAILY ACCOUNT some data 1 some data 2 some data 3 some data n END OF DAILY ACCOUNT START OF LEDGER BALANCE some data 1 some data 2 some data 3 some data 4 some data 5

Should we delete DataLake Analytic Job after completion?

阅读更多关于 Should we delete DataLake Analytic Job after completion?

问题 We are submitting U-SQL jobs very frequently and we a see a list of jobs previously submitted in ADLA. We see the total storage utilization of Data Lake store is increasing day by day. All of our jobs submitted only update one single output file and size is around 10 MB. The current storage utilization of Data Lake store is 9.3 GB. We think it's due to the previous jobs resources are still saved in the Data Lake. Should we take care of this or we should do something here? 回答1: I think the job

Using Azure Functions to call REST API and save results in Azure Data Lake gen2

阅读更多关于 Using Azure Functions to call REST API and save results in Azure Data Lake gen2

问题 I want to call a rest api and save the results as a csv or json file in Azure Data Lake Gen2. Based on what I have read Azure Functions is the way to go. The webservice returns data like the following format: "ID","ProductName","Company" "1","Apples","Alfreds futterkiste" "2","Oranges","Alfreds futterkiste" "3","Bananas","Alfreds futterkiste" "4","Salad","Alfreds futterkiste" ...next rows I have written a console app in C# which at the moment outputs the data to a console. The webservice uses

Value too long failure when attempting to convert column data

阅读更多关于 Value too long failure when attempting to convert column data

问题 Scenario I have a source file that contains blocks of JSON on each new line. I then have a simple U-SQL extract as follows where [RawString] represents each new line in the file and the [FileName] is defined as a variable from the @SourceFile path. @BaseExtract = EXTRACT [RawString] string, [FileName] string FROM @SourceFile USING Extractors.Text(delimiter:'\b', quoting : false); This executes without failure for the majority of my data and I'm able to parse the [RawString] as JSON further

Azure Data flow taking mins to trigger next pipeline

阅读更多关于 Azure Data flow taking mins to trigger next pipeline

Azure Data factory transferring data in Db in 10 millisecond but the issue I am having is it is waiting for few mins to trigger next pipeline and that ends up with 40 mins all pipelines are taking less than 20 ms to transfer data. But somehow it is waiting a few mins to trigger the next one. I used debug mode as well trigger the ADF using Logic App without debugging mood. Is there any way I can optimize it we want to move from SSIS to Data Flow but having a time issue 40 mins are so much in next step we have millions of records so it took 7 seconds to transfer data to dataBase but it waited

Upload data to the Azure ADLS Gen2 from on-premise using Python or Java

阅读更多关于 Upload data to the Azure ADLS Gen2 from on-premise using Python or Java

问题 I have an Azure Storage account with Data Lake Gen2. I would like to upload data from on-premise to the Lake Gen2 file systems using Python (or Java). I have found examples on how to interact with File Shares in the Storage account, yet I could not yet find out how to upload to the Lake (instead of the File Share). I have also found out how to do it for Gen1 Lakes here, but nothing except closed requests for Gen2. My question is whether this is even possible with Python as of today;

Should we delete DataLake Analytic Job after completion?

阅读更多关于 Should we delete DataLake Analytic Job after completion?

We are submitting U-SQL jobs very frequently and we a see a list of jobs previously submitted in ADLA. We see the total storage utilization of Data Lake store is increasing day by day. All of our jobs submitted only update one single output file and size is around 10 MB. The current storage utilization of Data Lake store is 9.3 GB. We think it's due to the previous jobs resources are still saved in the Data Lake. Should we take care of this or we should do something here? I think the job data expires after a couple of weeks, but if you are concerned and do not need the data for auditing or

Stream Analytics Job -> DataLake ouput

阅读更多关于 Stream Analytics Job -> DataLake ouput

I want to set up CI/CD (ARM template) with StreamAnalytics Job with output set to DataLake Store. https://docs.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs/outputs#microsoftdatalakeaccounts The issue comes with refreshToken: "It is recommended to put a dummy string value here when creating the data source and then going to the Azure Portal to authenticate the data source which will update this property with a valid refresh token" Furthermore after 90-days refresh token is outdated and you need to do "Renvew Authorization" https://docs.microsoft.com/en-us/azure

Azure Function exception could not load ActiveDirectory file or assembly

阅读更多关于 Azure Function exception could not load ActiveDirectory file or assembly

I am trying to write an azure timer function that writes files to Azure Datalake, but when I am adding the needed nuGet packages I am getting an error when I start the host The error is as follows [21/5/2018 8:36:20 AM] Executed 'NWPimFeederFromAws' (Failed, Id=03395101-41a5-44ef-96d8-f69c5d73eca7) [21/5/2018 8:36:20 AM] System.Private.CoreLib: Exception while executing function: NWPimFeederFromAws. NWPimFeeder: Could not load file or assembly 'Microsoft.IdentityModel.Clients.ActiveDirectory, Version=3.19.5.13701, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Could not find or load a