etl

I have series of ~10 queries to be executed every hour automatically in Redshift (maybe report success/failure). Most queries are aggregation on my tables. I have tried using AWS Lambda with CloudWatch Events , but Lambda functions only survive for 5 minutes max and my queries can take up to 25 minutes. systemjack It's kind of strange that AWS doesn't provide a simple distributed cron style service. It would be useful for so many things. There is SWF , but the timing/scheduling aspect is left up to the user. You could use Lambda/Cloudwatch to trigger SWF events. That's a lot of overhead to get

I Have to Upload Data in SQL Server from .dbf Files through SSIS. My Output Column is fixed but the input column is not fixed because the files come from client and client may have updated data by his own style. there may be some unused column too or input column name can be different from output column. One idea I had in my mind was to map files input column with output column in SQL Database table and use only those column which is present in the row for file id. But I am not getting how to do that. can you suggest me for doing the same or else you have any idea? Table Example. +--------+---

There is a requirement to extract source , destination and column names of source and destination . Why am I trying to do this is because I have thousands of packages and opening each package has on an average 60 to 75 of columns and listing all required info will take huge amount of time and its not a single time requirement and this task is done manually every two months in my organization currently. I'm looking for some ways to reverse engineer keeping all packages in a single folder and then go through each package and get the info and put it in some spreadsheet. I thought of opening

Recently, I have a project need to automate a process by combining SSIS package and excel VBA macro into one. Below are the steps: I have a SSIS package exporting all the view result to multiple individual file from sql server to excel. All the files are saved in same location. I have one excel VBA macro perform cleaning to remove all the empty sheets in each exported excel file. I also have a excel VBA macro perform merging task to merge all the excel file into in master excel file. This master excel file contains all the result set and each result set saved on different tabs accordingly.

I have click stream data such as referring URL, top landing pages, top exit pages and metrics such as page views, number of visits, bounces all in Google Analytics. There is no database yet where all this information might be stored. I am required to build a data warehouse from scratch(which I believe is known as web-house) from this data.So I need to extract data from Google Analytics and load it into a warehouse on a daily automated basis. My questions are:- 1)Is it possible? Every day data increases (some in terms of metrics or measures such as visits and some in terms of new referring