What is the easiest way to pull data from a blob and load it into a table in SQL Server?

半城伤御伤魂 提交于 2021-02-19 09:07:37

问题


I have hundreds of zipped files sitting in different folders, which I can access using MS Storage Explorer. I just setup a SQL Server DB in Azure. Now I am trying to figure out how I can pull data from each file in each folder, unzip it, parse it, and load it into tables. The data is coming in daily, so the folders are named '1', '2', '3', etc. '31', for the days of the month. Also, I have monthly folders '1' through '12', for the 12 months of the year. Finally, I have folders named '2017', '2018', and '2019'. I could ask some people on my team to write Python code to do the work, but it seems like there should be an easier way. I would like to avoid writing thousands of lines of Python code, if there is a simpler way to do this kind of thing. TIA.


回答1:


I would create Azure Logic App that would

  1. HTTP trigger (this will be manually ran)
  2. Iterate through all blobs present on storage account
  3. For each element use blob connector which has action called Extract archive to extract zipped files into blob
  4. Use Azure SQL BCP to pull this, if this is different format than CSV then use Azure Data Factory

Later if you expect new blobs do similar flow except use New or modified blob trigger.

For Azure SQL BCP from BLOB example with Logic Apps check this article https://marczak.io/posts/azure-loading-csv-to-sql/

And if you want general intro into Azure Logic Apps check this video https://youtu.be/ZvsOzji_8ow In here there is also new blob trigger demo.

In case you need data factory feel free to check this video. It has blob to sql demo too. Also you can trigger data factory from logic apps too. https://youtu.be/EpDkxTHAhOs



来源:https://stackoverflow.com/questions/57402746/what-is-the-easiest-way-to-pull-data-from-a-blob-and-load-it-into-a-table-in-sql

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!