Hi Michael,
We are currently on Finastra PhoenixEFE and currently retain copies of the end-of-month database and have been doing so since we were on Fiserv Premier. Thus, we have data going back to 2009. We are currently working to load all of the data in those databases into Snowflake.
When we move to SilverLake, we will be taking the ODS (operational data store) that we will be receiving on a daily basis and loading those tables into Snowflake (into what we call a landing zone) then using ETL to transform the data to standardize it to match what we’ve loaded from Phoenix. We currently load and replace our data within Snowflake and are working to enhance it to append. We can be inefficient because our data store is small. I just spoke to a $20B bank that rebuilds their data within Snowflake on a daily basis and their process takes just 30min (load to transform). Our process takes probably 10min and it’s single-threaded so we’re working to add parallel processing to the workflow to speed the process.
The goal is to have time dimensional data for all of our customer and accounts going back as far as possible and eventually update it in near real-time.