Benefits of using Staging Database while designing Data Warehouse

Question

I am in process of designing a Data Warehouse Architecture. While exploring various options to Extract data from Production and putting into Data Warehouse, I came across many a

Answer 1

There are a few potential advantages of using an intermediary staging database, which may or may not apply to your situation. There is no perfect, one-size fits all solution. Some of the potential advantages include:

• If it is appropriate, you can take a snapshot of your production database (you may have a daily backup or hot-site snapshot already) and then do your ETL from the restored backup or snapshot. This could save load on your production database.
• You may need complicated processing for your ETL which requires many intermediate tables which have no use except for the ETL process. You may not want to clutter your data warehouse with these intermediary tables.
• Your raw data may not be available all at once and you need somewhere to accumulate it before starting your ETL process to build your data warehouse.
• Your data warehouse may have production window requirements which can't be met by your ETL and so you need to stage your "output" (i.e. new records for the data warehouse) rather than or in addition to your production database.
• The production system may be in a highly secured environment and for whatever reason a decision may have been made to not allow the ETL process full access to the raw production data. The group that controls the production database may want to extract only the necessary data to a staging database so that the ETL process can only see what it needs. I've seen this where the production system and ETL process are managed by different third-party vendors.
• It may be that your ETL process creates large intermediate tables. Sometimes space management is easier if you start with an empty model database for your ETL staging area and then "throw it away" each day rather than trying to recover the space in a more surgical way, as you might do with a production or reporting database.

There are possible disadvantages too, which may or may not matter to you. Chief among these is having to have another database server. A lot of the advantages could be meaningless if you are using the same server to host the production and/or data warehouse databases.