Middleware ETL with CDC based on Event Driven Programming

Abstract

To achieve the real time data warehouse is strongly influenced by the process in the data warehouse known as Extract, Tranform, Loading (ETL). One way to optimize the ETL process is processing only data that undergoes changes on the On Line Transaction Processing (OLTP) system. This technique is known as Change Data Capture (CDC) which is designed to maximize the efficiency of the ETL process. In this research middleware was built as a place where the ETL process will be carried out, transaction data from the OLTP system will be captured and sent directly to the middleware for further processing. The method used to capture changes in OLTP systems is Change Data Capture (CDC) based on Event Driven Programming, where this technique relies on events that occur in OLTP in capturing data changes. Functional testing is done by making a simulation of the insert and update processes in test applications namely OLTP CRM system.The results of the research obtained are (1) Change Data Capture (CDC) based on Event Driven Programming can capture changes in data that occur in OLTP CRM database;(2) ETL process to load data from Normalized Data Store (NDS ) to data warehouse with Timestamp technique can load data that only undergoes changes that are processed to be loaded in Data Warehouse;(3) An increase in the amount of data that is processed has an effect on increasing processing time. Other factors that affect the value of process time are execution plan and cache memory