etl | 易学教程

Reading Huge volume of data from Sqlite to SQL Server fails at pre-execute

阅读更多关于 Reading Huge volume of data from Sqlite to SQL Server fails at pre-execute

问题 I have a huge (26GB) sqlite database that I want to import to SQL Server with SSIS. I have everything setup correctly. Some of the data flows are working correctly and importing the data. Data flows are simple. They just consist of source and destination. But when it comes to a table that has 80 million rows, data flow fails with this unhelpful message: Code: 0xC0047062 Source: Data Flow Task Source 9 - nibrs_bias_motivation [55] Description: System.Data.Odbc.OdbcException (0x80131937): ERROR

How do I split flat file data and load into parent-child tables in database?

阅读更多关于 How do I split flat file data and load into parent-child tables in database?

问题 I have denormalized data (coming from a file) that needs to be imported into parent-child tables. The source data is something like this: Account# Name Membership Email 101 J Burns Gold alpha@foo.com 101 J Burns Gold bravo@foo.com 101 J Burns Gold charlie@yay.com 227 H Gordon Silver red@color.com 350 B Clyde Silver italian@food.com 350 B Clyde Silver mexican@food.com What are the pieces, parts, or tactics of SSIS I should use to read the first three columns into a parent table, and the 4th

Why does my ODBC connection fail when running an SSIS load in Visual Studio but not when running the same package using Execute Package Utility

阅读更多关于 Why does my ODBC connection fail when running an SSIS load in Visual Studio but not when running the same package using Execute Package Utility

问题 I'm working on a Data Mart loading package in SSIS 2012. When attempting to execute the package in Visual Studio I get this error: "The AcquireConnection method call to the connection manager Data Warehouse.ssusr failed with error code 0xC0014009". When I test the connectivity of the Connection Manager Data Warehouse.ssusr I see that it passes. When I execute the package outside of Visual Studio using the Execute Package Utility, the package runs. I don't understand what's going on. The

Automate Version number Retrieval from .Dtsx files

阅读更多关于 Automate Version number Retrieval from .Dtsx files

问题 Is there a way or query by which I can find the version number of SSIS packages (*.dtsx files)? I have the *.dtsx files in my Team Foundation Server for which I wanted to know. The manual way is to do a mouse right-click on the package and click Compare to see the VersionBuild but there are like thousands of packages so doing it manually is really not possible Note: The process should be automated, not manual 回答1: Getting values within dtsx packages If you are trying to read a package version

convert Excel Date Serial Number to Regular Date

阅读更多关于 convert Excel Date Serial Number to Regular Date

问题 I got a column called DateOfBirth in my csv file with Excel Date Serial Number Date Example: 36464 37104 35412 When i formatted cells in excel these are converted as 36464 => 1/11/1999 37104 => 1/08/2001 35412 => 13/12/1996 I need to do this transformation in SSIS or in SQL. How can this be achieved? 回答1: In SQL: select dateadd(d,36464,'1899-12-30') -- or thanks to rcdmk select CAST(36464 - 2 as SmallDateTime) In SSIS, see here http://msdn.microsoft.com/en-us/library/ms141719.aspx 回答2: The

[SQL]~ETL讲解

阅读更多关于 [SQL]~ETL讲解

ETL讲解（很详细！！！） ETL是将业务系统的数据经过抽取、清洗转换之后加载到数据仓库的过程，目的是将企业中的分散、零乱、标准不统一的数据整合到一起，为企业的决策提供分析依据。 ETL是BI项目重要的一个环节。通常情况下，在BI项目中ETL会花掉整个项目至少1/3的时间,ETL设计的好坏直接关接到BI项目的成败。　　ETL的设计分三部分：数据抽取、数据的清洗转换、数据的加载。在设计ETL的时候我们也是从这三部分出发。数据的抽取是从各个不同的数据源抽取到ODS(Operational Data Store，操作型数据存储)中——这个过程也可以做一些数据的清洗和转换)，在抽取的过程中需要挑选不同的抽取方法，尽可能的提高ETL的运行效率。ETL三个部分中，花费时间最长的是“T”(Transform，清洗、转换)的部分，一般情况下这部分工作量是整个ETL的2/3。数据的加载一般在数据清洗完了之后直接写入DW(Data Warehousing，数据仓库)中去。　　ETL的实现有多种方法，常用的有三种。一种是借助ETL工具(如Oracle的OWB、SQL Server 2000的DTS、SQL Server2005的SSIS服务、Informatic等)实现，一种是SQL方式实现，另外一种是ETL工具和SQL相结合。前两种方法各有各的优缺点，借助工具可以快速的建立起ETL工程

How to load data from SQL Server to SAP BW using SSIS

阅读更多关于 How to load data from SQL Server to SAP BW using SSIS

问题 I would like to load data from SQL server to SAP BW using SSIS. Could some one help me on it how can i do it. Currently i am using sql server 2005. 回答1: Why would you use SSIS for that? I would recommend either load using SAP BW standard anyDB source system or using BO Data Services. Both do it natively and well. If you insist, look at: https://theobald-software.com/en/xtract-is-productinfo.html They have the following feature: Xtract IS BW Loader With the Xtract IS BW Loader data target, you

OutOfMemoryException while trying to read big Excel file into DataTable

阅读更多关于 OutOfMemoryException while trying to read big Excel file into DataTable

问题 I'm using SSIS package to clean and load data from .Xlsx file to SQL Server table. I have also to highlight cells containing wrong data in .Xlsx file, for this I have to get back column and row indexes based on column name and row id(witch I have in my data spreadsheet). For that I compare each column name from my first spreadsheet ( Error_Sheet ) with rows of a column that I added in a second spreadsheet and do the same for rows, and if I have the same value of cells I get back the column

How to traverse multiple folders & multiple files & dump data to SQL tables with same filename? SSIS

阅读更多关于 How to traverse multiple folders & multiple files & dump data to SQL tables with same filename? SSIS

问题 This will be a bit of an update from the question I asked here before. I need to traverse folders & dump txt files to SQL tables with the same name (barring the .txt extension) My folder/file structure is setup as shown below Now the FileA,B,C are consistent throughout all the folders & there happens to be a [dbo].[FileA],[dbo].[FileB],[dbo].[FileC] etc sitting on the server. So Data1 Folder will have FileA,FileB...FileZ & so will Data200 Folder. The goal is to traverse through all the

Find lowest and highest values split into rows from a single string of concatenated values

阅读更多关于 Find lowest and highest values split into rows from a single string of concatenated values

问题 This is a follow-up to my question here: in which I got an excellent answer for that question provided by uzi . I however noticed that a new Company, Company3 also used single data Points, such as account 6000 which does not follow the manner of the previous companies which makes uzi's recursive cte not applicable. As such I feel like it is required to alter the question, but I Believe that this complication would issue a new question rather than an edit on my previous one due to having a