u-sql

I'm considering using Data Lake technologies which I have been studying for the latest weeks, compared with the traditional ETL SSIS scenarios, which I have been working with for so many years. I think of Data Lake as something very linked to big data, but where is the line between using Data Lake technolgies vs SSIS? Is there any advantage of using Data Lake technologies with 25MB ~100MB ~ 300MB files? Parallelism? flexibility? Extensible in the future? Is there any performance gain when the files to be loaded are not so big as U-SQL best scenario... What are your thoughts? Would it be like

I'm trying to parse below Json file using USQL but keep getting error. Json file@ {"dimBetType_SKey":1,"BetType_BKey":1,"BetTypeName":"Test1"} {"dimBetType_SKey":2,"BetType_BKey":2,"BetTypeName":"Test2"} {"dimBetType_SKey":3,"BetType_BKey":3,"BetTypeName":"Test3"} Below is the USQL script, I'm trying to extract the data from above file. REFERENCE ASSEMBLY [Newtonsoft.Json]; REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats]; DECLARE @Full_Path string = "adl://xxxx.azuredatalakestore.net/2017/03/28/00_0_66ffdd26541742fab57139e95080e704.json"; DECLARE @Output_Path = "adl://xxxx

I have custom extractor, and I'm trying to log some messages from it. I've tried obvious things like Console.WriteLine , but cannot find where output is. However, I found some system logs in adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/ . How can I log something? Is it possible to specify log file somewhere on Data Lake Store or Blob Storage Account? A recent release of U-SQL has added diagnostic logging for UDOs. See the release notes here . // Enable the diagnostics preview feature SET @@FeaturePreviews = "DIAGNOSTICS:ON"; // Extract as one column @input

Presently, I have all my data files in Azure Data Lake Store. I need to process these files which are mostly in csv format. The processing would be running jobs on these files to extract various information for e.g.Data for certain periods of dates or certain events related to a scenario or adding data from multiple tables/files. These jobs run everyday through u-sql jobs in data factory(v1 or v2) and then sent to powerBI for visualization. Using ADLA for all this processing, I feel it takes a lot of time to process and seems very expensive. I got a suggestion that I should use Azure

I am working with integrating continous integration with vsts using MSBuild for U-SQL reffering the link : https://blogs.msdn.microsoft.com/azuredatalake/2017/10/24/continuous-integration-made-easy-with-msbuild-support-for-u-sql-preview/ But i am facing below issue while working with VSTS Build server Source\TrainingUsql\TrainingUsql\TrainingUsql\TrainingUsql.usqlproj (0, 0) Source\TrainingUsql\TrainingUsql\TrainingUsql\TrainingUsql.usqlproj(0,0): Error MSB4057: The target "Build" does not exist in the project. Process 'msbuild.exe' exited with code '1'. I have followed all the steps given in

I'm trying to build CI/CD for Azure Data lake analytics - USQL code and when i build the code using Visual studio build option in VSTS getting the below error - Using the Private agent for taking the build - C:\Users\a.sivananthan\AppData\Roaming\Microsoft\DataLake\MsBuild\1.0\Usql.targets(33,5): Error MSB4062: The "Microsoft.Cosmos.ScopeStudio.VsExtension.CompilerTask.USqlCompilerTask" task could not be loaded from the assembly Microsoft.Cosmos.ScopeStudio.VsExtension.CompilerTask. Could not load file or assembly 'Microsoft.Cosmos.ScopeStudio.VsExtension.CompilerTask' or one of its

Is is possible to implement Loops (while/for) in U-SQL without using C#. If no, can anyone share the c# syntax to implement loops in u-sql. I am extracting files from a particular date to a date, but right now I am extracting this by writing file path manually. DROP VIEW IF EXISTS dbo.ReadingConsolidated; CREATE VIEW IF NOT EXISTS dbo.ReadingConsolidated AS EXTRACT ControllerID int?, sensorID int?, MeasureDate DateTime, Value float FROM "adl://datalake.azuredatalakestore.net/2015/7/1/Reading.csv", "adl://datalake.azuredatalakestore.net/2015/7/2/Reading.csv", "adl://datalake.azuredatalakestore