azure-data-lake

U- SQL Unable to extract data from JSON file

我与影子孤独终老i 提交于 2019-11-28 08:03:48
问题 I was trying to extract data from a JSON file using USQL. Either the query runs successfully without producing any output data or results in "vertex failed fast error". The JSON file looks like: { "results": [ { "name": "Sales/Account", "id": "7367e3f2-e1a5-11e5-80e8-0933ecd4cd8c", "deviceName": "HP", "deviceModel": "g6-pavilion", "clientip": "0.41.4.1" }, { "name": "Sales/Account", "id": "c01efba0-e0d5-11e5-ae20-af6dc1f2c036", "deviceName": "acer", "deviceModel": "veriton", "clientip": "10

30Mb limit uploading to Azure DataLake using DataLakeStoreFileSystemManagementClient

别等时光非礼了梦想. 提交于 2019-11-28 07:42:59
问题 I am receiving an error when using _adlsFileSystemClient.FileSystem.Create(_adlsAccountName, destFilePath, stream, overwrite) to upload files to a datalake. The error comes up with files over 30Mb. It works fine with smaller files. The error is: at Microsoft.Azure.Management.DataLake.Store.FileSystemOperations.d__16.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System

U-SQL - Extract data from json-array

守給你的承諾、 提交于 2019-11-27 21:42:00
Already tried the suggested JSONPath option, but it seems the JSONExtractor only recognizes root level. In my case I have to deal with a nested json-structure, with an array as well (see example below). Any options for extracting this without multiple intermediate files? "relation": { "relationid": "123456", "name": "relation1", "addresses": { "address": [{ "addressid": "1", "street": "Street 1", "postcode": "1234 AB", "city": "City 1" }, { "addressid": "2", "street": "Street 2", "postcode": "5678 CD", "city": "City 2" }] }} SELECT relationid, addressid, street, postcode, city ? Michael Rys

U-SQL Output in Azure Data Lake

柔情痞子 提交于 2019-11-27 09:08:39
Would it be possible to automatically split a table into several files based on column values if I don't know how many different key values the table contains? Is it possible to put the key value into the filename? Michael Rys This is our top ask (and has been previously asked on stackoverflow too :). We are currently working on it and hopefully have it available by summer. Until then you have to write a script generator. I tend to use U-SQL to generate the script but you could do it with Powershell or T4 etc. Here is an example: Let's assume you want to write files for the column name in the

U-SQL - Extract data from json-array

纵然是瞬间 提交于 2019-11-26 20:46:53
问题 Already tried the suggested JSONPath option, but it seems the JSONExtractor only recognizes root level. In my case I have to deal with a nested json-structure, with an array as well (see example below). Any options for extracting this without multiple intermediate files? "relation": { "relationid": "123456", "name": "relation1", "addresses": { "address": [{ "addressid": "1", "street": "Street 1", "postcode": "1234 AB", "city": "City 1" }, { "addressid": "2", "street": "Street 2", "postcode":

U-SQL Output in Azure Data Lake

守給你的承諾、 提交于 2019-11-26 14:25:26
问题 Would it be possible to automatically split a table into several files based on column values if I don't know how many different key values the table contains? Is it possible to put the key value into the filename? 回答1: This is our top ask (and has been previously asked on stackoverflow too :). We are currently working on it and hopefully have it available by summer. Until then you have to write a script generator. I tend to use U-SQL to generate the script but you could do it with Powershell