etl

SSIS Source Format Implicit Conversion for Datetime

别等时光非礼了梦想. 提交于 2019-11-27 05:37:33
Does anyone know of a chart that shows which formats of dates are supported by the different datatypes in SSIS? For example, does DT_DBTimestamp support the format 1-Jan . I tried looking but couldn't find anything. The reason I ask is that I found that when I try to do a Convert(datetime, '1-Jan') it fails which I expected. However, when I pull this same value into SSIS into the DT_DBTimestamp , it will implicitly identify it as 1/1/2017 rather than redirect the row as a datatype conversion error. General Info These are the default formats of datetimes datatypes (when converting from string)

How to extract data from Google Analytics and build a data warehouse (webhouse) from it?

与世无争的帅哥 提交于 2019-11-27 05:20:01
问题 I have click stream data such as referring URL, top landing pages, top exit pages and metrics such as page views, number of visits, bounces all in Google Analytics. There is no database yet where all this information might be stored. I am required to build a data warehouse from scratch(which I believe is known as web-house) from this data.So I need to extract data from Google Analytics and load it into a warehouse on a daily automated basis. My questions are:- 1)Is it possible? Every day data

alter table then update in single statement

懵懂的女人 提交于 2019-11-27 03:12:01
问题 I have a requirement where I need to Alter (Add 2 columns) and then update the same table. Here is the query I tried: ALTER TABLE A ADD c1 int,c2 varchar(10) UPDATE A set c1 = 23, c2 = 'ZZXX' I need to run the above two queries at a time. I am using Talend ETL tool, in this we have a component tMssqlrow, which allow us to run multiple queries (I am using 10 to 15 update queries in single component). But the above query is not working. I tested in DataBase Microsoft SQL. i am getting the below

Importing excel files having variable headers

孤者浪人 提交于 2019-11-27 02:15:32
I have the SSIS package, which will load the excel file into Database. I have created Excel Source task to map the excel column name to Database table column name and its working fine. In rare case, We are receiving the excel file column name with some space (for example : Column name is "ABC" but we are receiving "ABC ") and which cause the mapping issue and SSIS got failed. Is there any possible to trim the column name without opening the excel. Note : Page name will be dynamic and Column position may change (eg: Column "ABC may exist in first row or second row or .."). First of all, my

Automate Version number Retrieval from .Dtsx files

荒凉一梦 提交于 2019-11-27 02:08:46
Is there a way or query by which I can find the version number of SSIS packages (*.dtsx files)? I have the *.dtsx files in my Team Foundation Server for which I wanted to know. The manual way is to do a mouse right-click on the package and click Compare to see the VersionBuild but there are like thousands of packages so doing it manually is really not possible Note: The process should be automated, not manual Getting values within dtsx packages If you are trying to read a package version within this package you can access to one of the SSIS system variables Variable Type Description ----------

How to set dependencies between DAGs in Airflow?

安稳与你 提交于 2019-11-27 00:21:59
问题 I am using Airflow to schedule batch jobs. I have one DAG (A) that runs every night and another DAG (B) that runs once per month. B depends on A having completed successfully. However B takes a long time to run and so I would like to keep it in a separate DAG to allow better SLA reporting. How can I make running DAG B dependent on a successful run of DAG A on the same day? 回答1: You can achieve this behavior using an operator called ExternalTaskSensor. Your task (B1) in DAG(B) will be

SSIS - How to access a RecordSet variable inside a Script Task

隐身守侯 提交于 2019-11-26 23:19:18
问题 How do you access a RecordSet variable inside a Script Task? 回答1: On the script tab, make sure you put the variable in either the readonlyvariables or readwritevariables text boxes. Here is a simple script that I use to format the errors in a data flow (saved in a RecordSet Variable) into the body of an email. Basically I read the recordset varialbe into a datatable and process it row by row with the for loops. After this task completes I examine the value of uvErrorEmailNeeded to determine

Using Pentaho Kettle, how do I load multiple tables from a single table while keeping referential integrity?

橙三吉。 提交于 2019-11-26 22:50:47
问题 Need to load data from a single file with a 100,000+ records into multiple tables on MySQL maintaining the relationships defined in the file/tables; meaning the relationships already match. The solution should work on the latest version of MySQL, and needs to use the InnoDB engine; MyISAM does not support foreign keys. I am a completely new to using Pentaho Data Integration (aka Kettle) and any pointers would be appreciated. I might add that it is a requirement that the foreign key

Copy a few of the columns of a csv file into a table

北战南征 提交于 2019-11-26 21:58:31
I have a CSV file with 10 columns. After creating a PostgreSQL table with 4 columns, I want to copy some of 10 columns into the table. the columns of my CSV table are like: x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 the columns of my PostgreSQL table should be like: x2 x5 x7 x10 If it is an ad hoc task Create a temporary table with all the columns in the input file create temporary table t (x1 integer, ... , x10 text) Copy from the file into it: copy t (x1, ... , x10) from '/path/to/my_file' with (format csv) Now insert into the definitive table from the temp: insert into my_table (x2, x5, x7, x10) select

How to Map Input and Output Columns dynamically in SSIS?

雨燕双飞 提交于 2019-11-26 21:47:47
问题 I Have to Upload Data in SQL Server from .dbf Files through SSIS. My Output Column is fixed but the input column is not fixed because the files come from client and client may have updated data by his own style. there may be some unused column too or input column name can be different from output column. One idea I had in my mind was to map files input column with output column in SQL Database table and use only those column which is present in the row for file id. But I am not getting how to