etl

Is there a direct route that is pretty straight forward? (i.e. can SQL Server read XML) Or, is it best to parse the XML and just transfer it in the usual way via ADO.Net either as individual rows or perhaps a batch update? I realize there may be solutions that involve large complex stored procs--while I'm not entirely opposed to this, I tend to prefer to have most of my business logic in the C# code. I have seen a solution using SQLXMLBulkLoad, but it seemed to require fairly complex SQL code. For reference, I'll be working with about 100 rows at a time with about 50 small pieces of data for

I have a CSV file with a {LF} delimiting each row and a date column with the date format as "12/20/2010" (including quotation marks) My destination column is a SQL Server 2008 database table of type date (not datetime) In my Flat File Connection Manager, I have configured the date column to be data type date [DT_DATE] with TextQualified set to true and the column delimiter as {LF} (it is the last column on each row). I have the text qualifier set to " When I try to load this into an OLE Destination I get the following error [TRN_DORPS [760]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB

During an ETL process I needed to extract and load a JSON column from one Postgres database to another. We use Pandas for this since it has so many ways to read and write data from different sources/destinations and all the transformations can be written using Python and Pandas. We're quite happy with the approach to be honest.. but we hit a problem. Usually it's quite easy to read and write the data. You just use pandas.read_sql_table to read the data from the source and pandas.to_sql to write it to the destination. But, since one of the source tables had a column of type JSON (from Postgres)

主花了一下午时间，收集全网，学习了下kittle,觉得该大v写的不错，特意整理给大家！学会这几个案例kittle基本就没问题了。 1.kettle案例一抽取gz格式文本内容保存到mongodb https://blog.csdn.net/zzq900503/article/details/78658649 2.kettle案例二抽取mysql数据保存到mongodb https://blog.csdn.net/zzq900503/article/details/78678030 3.kettle案例三抽取json数据保存到mongodb https://blog.csdn.net/zzq900503/article/details/78745397 4.kettle案例四使用java脚本进行数据处理 https://blog.csdn.net/zzq900503/article/details/78745359 5.kettle案例五使用js脚本进行json数据处理新增行 https://blog.csdn.net/zzq900503/article/details/78952318 6.kettle案例六数据表关联--排序记录-记录集连接-过滤记录 https://blog.csdn.net/zzq900503/article/details/78962029 7

I have recently been exposed to some ETL tools such as Talend and Apatar and I was wondering what exactly the purpose/main goal of these tools is in laymans terms. Who primarily uses them and if you use them, how they are (from my understanding) better than just writing some type of scripts. ETL stands for "Extract/Transform/Load". These tools take data from one source and move it into another. You can map schemas from the source to the destination in unique ways, transform and cleanse data before it moves into the destination, and load the destination in an efficient manner. You can schedule

I have a few SSIS packages that were password-protected (their protection level is apparently EncryptAllWithPassword) by a developer who left the company and can't be reached anymore, and trying to open them gives the following error since the password can't be supplied: Error loading 'Package.dtsx' : Failed to remove package protection with error 0xC0014037 "The package is encrypted with a password. The password was not specified, or is not correct.". This occurs in the CPackage::LoadFromXML method. Is there any any way to open these packages? I have access to the administrator account