azure-data-lake

来源： https://stackoverflow.com/questions/48165947/access-datalake-from-azure-datafactory-v2-using-on-demand-hd-insight-cluster

问题 During a training script executed on a compute target, we're trying to download a registered Dataset from an ADLS2 Datastore. The problem is that it takes hours to download ~1.5Gb (splitted into ~8500 files) to the compute target with the following method : from azureml.core import Datastore, Dataset, Run, Workspace # Retrieve the run context to get Workspace RUN = Run.get_context(allow_offline=True) # Retrieve the workspace ws = RUN.experiment.workspace # Creating the Dataset object based on

问题 I have to load data lake file(csv format) to azure datalake storage gen2 using logic app.I have created logic app using http action,able to create the file and appended the data.for the next http action need to give the length.what is content-type to be used for files to load data into datalake storage gen2.i'm getting error like The uploaded data is not contiguous or the position query parameter value is not equal to the length of the file after appending the uploaded data and errocode:

问题 I had been reading about spark predicates pushdown and partition pruning to understand the amount of data read. I had the following doubts related to the same Suppose I have a dataset with columns (Year: Int, SchoolName: String, StudentId: Int, SubjectEnrolled: String) of which the data stored on disk is partitioned by Year and SchoolName and stored in parquet format at say azure data lake storage. 1) If I issue a read spark.read(container).filter(Year=2019, SchoolName="XYZ"): Will Partition

问题 I have ioT data in azure datalake structure as {date}/{month}/{day}/abbs. Json Each file has multiple records separated by new line .. How to read this data using usql and load into table and query. When I load it in usql table using / / / / .json will that load data into same table when new files added to files. I have followed qzure docs but did not find any answer to line separated json file. 回答1: In this example we will create a table to store events: CREATE TABLE dbo.Events ( Event

问题 All I want to do, is to upload files from on prime to Azure Data Lake Storage Gen2 using the Azure CLI (via ` command), but have a connection error! Can I use Azure CLI to to that? Or I have to use another tool? PS: I cannot use Azure Data Factory, I want my job running from my on prime and not from the cloud! Thks. azure.datalake.store.exceptions.DatalakeRESTException: HTTP error: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='storageAccount.azuredatalakestore.net', port=443): Max

问题 One of the team member after upgrading the ADL tools for VS to version 2.3.4000.x, getting the below error.. Error : (-1,-1) 'E_CSC_SYSTEM_INTERNAL: Internal error! The ObjectManager found an invalid number of fixups. This usually indicates a problem in the Formatter.' Compile failed! Tried to downgrade back to version ( 2.3.3000.2 ), it didn't help much. If encountered similar issue, found the reason and resolved it, please share it. 回答1: After trying out few unsuccessful options, decided to

问题 Copy activity using U-SQL can any body share script I want to read a file using copy activity using U-SQL and write this file on output in a file using U-SQL 回答1: You can copy files in the following ways: Azure Data Factory ADL Copy tool Write a custom U-SQL extractor/outputter (similar to 2) 回答2: Here's a UDO that does that. The other samples are also quite useful. https://github.com/Azure/usql/tree/master/Examples/FileCopyUDOs 来源： https://stackoverflow.com/questions/44048110/copy-activity

问题 Copy activity using U-SQL can any body share script I want to read a file using copy activity using U-SQL and write this file on output in a file using U-SQL 回答1: You can copy files in the following ways: Azure Data Factory ADL Copy tool Write a custom U-SQL extractor/outputter (similar to 2) 回答2: Here's a UDO that does that. The other samples are also quite useful. https://github.com/Azure/usql/tree/master/Examples/FileCopyUDOs 来源： https://stackoverflow.com/questions/44048110/copy-activity

问题 I try create a custom parallel extractor, but i have no idea how do it correctly. I have a big files (more than 250 MB), where data for each row are stored in 4 lines. One file row store data for one column. Is this possible to create working parallely extractor for large files? I am afraid that data for one row, will be in different extents after file splitting. Example: ... Data for first row Data for first row Data for first row Data for first row Data for second row Data for second row