etl

I have the following simple example: CREATE TABLE Cars ( Cars, Item, Value ) AS SELECT 'bmw', 'wheels', '4' FROM DUAL UNION ALL SELECT 'bmw', 'color', 'red' FROM DUAL UNION ALL SELECT 'bmw', 'price', '5' FROM DUAL UNION ALL SELECT 'mercedes', 'wheels', '4' FROM DUAL UNION ALL SELECT 'mercedes', 'color', 'black' FROM DUAL UNION ALL SELECT 'lambo', 'wheels', '5' FROM DUAL UNION ALL SELECT 'lambo', 'color', 'yellow' FROM DUAL UNION ALL SELECT 'lambo', 'price', '7' FROM DUAL UNION ALL SELECT 'mercedes', 'price', '6' FROM DUAL; The thing is that I need to "pivot" the table to get items as column

I have a huge (26GB) sqlite database that I want to import to SQL Server with SSIS. I have everything setup correctly. Some of the data flows are working correctly and importing the data. Data flows are simple. They just consist of source and destination. But when it comes to a table that has 80 million rows, data flow fails with this unhelpful message: Code: 0xC0047062 Source: Data Flow Task Source 9 - nibrs_bias_motivation [55] Description: System.Data.Odbc.OdbcException (0x80131937): ERROR [HY000] unknown error (7) at System.Data.Odbc.OdbcConnection.HandleError(OdbcHandle hrHandle, RetCode

In SSIS, I try to load data from a flat file. The flat file have fixed width columns, but some column are not present in a row (a column can have a CRLF, which must be a new line) like this a b c the first rowok<CRLF> iu jjrjdd<CRLF> this is a newline<CRLF> How I can have exactly the same number of line and exact data in my output ? I setup a flat file connection, of ragged right type. In this sample, row 1 is correctly retrieve, but for row 2, it didn't recognize CRLF, and put in b column all the 3rd row ... thanks in advance Regards, Xavier Workaround In the flat file connection manager read

I have denormalized data (coming from a file) that needs to be imported into parent-child tables. The source data is something like this: Account# Name Membership Email 101 J Burns Gold alpha@foo.com 101 J Burns Gold bravo@foo.com 101 J Burns Gold charlie@yay.com 227 H Gordon Silver red@color.com 350 B Clyde Silver italian@food.com 350 B Clyde Silver mexican@food.com What are the pieces, parts, or tactics of SSIS I should use to read the first three columns into a parent table, and the 4th column (Email) into a child table? I have several options for the parent key which I am permitted to take

How do you access a RecordSet variable inside a Script Task? On the script tab, make sure you put the variable in either the readonlyvariables or readwritevariables text boxes. Here is a simple script that I use to format the errors in a data flow (saved in a RecordSet Variable) into the body of an email. Basically I read the recordset varialbe into a datatable and process it row by row with the for loops. After this task completes I examine the value of uvErrorEmailNeeded to determine if there is anything to email using a conditional process flow connector. You will also need to add a

Need to load data from a single file with a 100,000+ records into multiple tables on MySQL maintaining the relationships defined in the file/tables; meaning the relationships already match. The solution should work on the latest version of MySQL, and needs to use the InnoDB engine; MyISAM does not support foreign keys. I am a completely new to using Pentaho Data Integration (aka Kettle) and any pointers would be appreciated. I might add that it is a requirement that the foreign key constraints are NOT disabled. Since it's my understanding that if there is something wrong with the database's

I'd like to convert result table to JSON array in MySQL using preferably only plain MySQL commands. For example with query SELECT name, phone FROM person; | name | phone | | Jack | 12345 | | John | 23455 | the expected JSON output would be [ { "name": "Jack", "phone": 12345 }, { "name": "John", "phone": 23455 } ] Is there way to do that in plain MySQL? EDIT: There are some answers how to do this with e.g. MySQL and PHP , but I couldn't find pure MySQL solution. New solution: Built using Your great comments, thanks! SELECT JSON_ARRAYAGG(JSON_OBJECT('name', name, 'phone', phone)) from Person;

一、数据仓库的架构 数据仓库（Data Warehouse DW）是为了便于多维分析和多角度展现而将数据按特定的模式进行存储所建立起来的关系型数据库，它的数据基于OLTP源系统。数据仓库中的数据是细节的、集成的、面向主题的，以OLAP系统的分析需求为目的。 数据仓库的架构模型包括了星型架构（图二：pic2.bmp）与雪花型架构（图三：pic3.bmp）两种模式。如图所示，星型架构的中间为事实表，四周为维度表，类似星星；而相比较而言，雪花型架构的中间为事实表，两边的维度表可以再有其关联子表，从而表达了清晰的维度层次关系。 从OLAP系统的分析需求和ETL的处理效率两方面来考虑：星型结构聚合快，分析效率高；而雪花型结构明确，便于与OLTP系统交互。因此，在实际项目中，我们将综合运用星型架构与雪花型架构来设计数据仓库。 那么，下面我们就来看一看，构建企业级数据仓库的流程。 二、构建企业级数据仓库五步法 （一）、确定主题 即确定数据分析或前端展现的主题。例如：我们希望分析某年某月某一地区的啤酒销售情况，这就是一个主题。主题要体现出某一方面的各分析角度（维度）和统计数值型数据（量度）之间的关系，确定主题时要综合考虑。 我们可以形象的将一个主题想象为一颗星星：统计数值型数据（量度）存在于星星中间的事实表；分析角度（维度）是星星的各个角；我们将通过维度的组合，来考察量度。那么，

using SSIS 2012. My flat file connection manager I have a delimited file where the row delimiter is set to CRLF , but when it processes the file, I have a text column that has an LF in it. This is causing it to read that as a row terminator causing it fail. Any ideas? Before answering, i don't think that the column contains only LF because if the row delimiter is CRLF it will not consider it as delimiter. So it is probably CRLF , but i will give a solution for the two cases (CRLF or LF) Solution You can fix this situation with the following steps: First in the Flat File connection manager add