azure-data-lake

问题 When trying to call my azure data lake store space and list all directories , It fails to return any results during to missing CORS .The problem that I can't find any way to give the Azure DatalakeStore the CORS policy ( Not the same as Azure Storage ) Any workaround or solution for this ? 回答1: Some of the services in Azure still doesn't have CORS support (like DocumentDB and as you have experienced seems for ADLS as well). One thing that can be tried before the same is provided by azure is

问题 I am processing data in U-SQL but not getting expected results. Here is what I am doing: 1- Select data from ADL table partitions and assign it to @data1 2- Aggregate data using Group BY and assign it to @data2 3- Truncate partitions 4- Insert data(produced in step 2) into the same table 5- Use @data2 and generate a unique GUID for every record using user defined function and assign it to @data2 //UDF Code public static Guid GetNewGuid () { return Guid.NewGuid (); } 6- Select few columns from

问题 We are moving the data from Azure Table Storage to Azure Data Lake Store with the help of Data Factory. As part of if I have added a linked service which connects to Azure Data Lake Store. { "name": "AzureDataLakeStoreLinkedService", "properties": { "description": "", "hubName": "xxxxxxxxxxxxxxxxxxxxxx", "type": "AzureDataLakeStore", "typeProperties": { "dataLakeStoreUri": "https://xxxxxxxxxxxxx.azuredatalakestore.net/webhdfs/v1", "authorization": "**********", "sessionId": "**********",

问题 I have a custom USql applier which extends the IApplier class. [SqlUserDefinedApplier] public class CsvApplier : IApplier { public CsvApplier() { //totalcount = count; } public override IEnumerable<IRow> Apply(IRow input, IUpdatableRow output) { //....custom logic //yield return or yield break } } This applier is then used from Usql script as @log = SELECT t.ultimateID, t.siteID, . . . t.eTime, t.hours FROM @logWithCount CROSS APPLY new BSWBigData.USQLApplier.CsvApplier() AS t(ultimateID

问题 Is it possible to write regular expression comparisons in USQL? For example, rather than multiple "LIKE" statements to search for the name of various food items, I want to perform a comparison of multiple items using a single Regex expression. 回答1: You can create a new Regex object inline and then use the IsMatch() method. The example below returns "Y" if the Offer_Desc column contains the word "bacon", "croissant", or "panini". @output = SELECT , CSHARP(new Regex("\\b(BACON|CROISSANT|PANINI

问题 how can we have dynamic output file name in u-sql in azure data lake based on timestamp when job is executed.Thanks for help.My code as below: OUTPUT @telDataResult TO @"wasb://blobcontainer@blobstorage.blob.core.windows.net/**yyyymmdd**_TelDataOutput.Csv" USING Outputters.Csv(); 回答1: This feature is currently in development but not available yet. Feel free to add your vote to the feature request: https://feedback.azure.com/forums/327234-data-lake/suggestions/10550388-support-dynamic-output

问题 What is the best way to send data from Event Hubs to Data Lake Store? 回答1: I am assuming you want to ingest data from EventHubs to Data Lake Store on a regular basis. Like Nava said, you can use Azure Stream Analytics to get data from EventHub into Azure Storage Blobs. Thereafter you can use Azure Data Factory (ADF) to copy data on a scheduled basis from Blobs to Azure Data Lake Store. More details on using ADF are available here: https://azure.microsoft.com/en-us/documentation/articles/data

问题 I'm considering using Data Lake technologies which I have been studying for the latest weeks, compared with the traditional ETL SSIS scenarios, which I have been working with for so many years. I think of Data Lake as something very linked to big data, but where is the line between using Data Lake technolgies vs SSIS? Is there any advantage of using Data Lake technologies with 25MB ~100MB ~ 300MB files? Parallelism? flexibility? Extensible in the future? Is there any performance gain when the

问题 I am making a custom activity in ADF, which involves reading multiple files from Azure Storage Blobs, doing some work on them, and then finally writing a resulting file to the Azure Data Lake Store. Last step is where I stop, because as far as I can see, the .NET SDK only allows for uploading from a local file. Is there any way to to (programmatically) upload a file to ADL Store, where it is not from a local file? Could be a blob or a stream. If not, any workarounds? 回答1: Yes, it's possible

问题 I am working on an IoT analytics solution which consumes Avro formatted messages fired at an Azure IoT Hub and (hopefully) uses Stream Analytics to store messages in Data Lake and blob storage. A key requirement is the Avro containers must appear exactly the same in storage as they did when presented to the IoT Hub, for the benefit of downstream consumers. I am running into a limitation in Stream Analytics with granular control over individual file creation. When setting up a new output