Implementation of Data Backup and Synchronization Based on Identity Column Real Time Data Warehouse
Abstract
Failure in the process of loading data from the Online Transactional Processing(OLTP) system to the Normalized Data Store (NDS) database can occur. This caused by a disruption in the network so that the OLTP system is unable to save data to the OLTP and NDS databases. Backup and synchronization data scenarios are needed to maintain data consistency and data availability. In this research, the process of data backup and synchronization is done by providing an identity column for the table in the OLTP database. An identity column is used to give status data, value '0' if the inserting process fails, and value '1' if successful. Data backup is done by storing temporary data into a CSV file format, then the CSV file is read, and an insert process is carried out into the OLTP database. After the insertion process into the OLTP database is successful, it continues with the synchronization process between the OLTP database and the NDS. Data synchronization between OLTP and NDS databases is done by checking the value of the Identity Column in each table in the OLTP database.
Downloads
References
[2] V. Rainardi, Building a data warehouse: With examples in SQL server. 2008.
[3] A. Khalaf Hamoud, A. Salah Hashim, and W. Akeel Awadh, “CLINICAL DATA WAREHOUSE A REVIEW,” Iraqi Journal Computing Informatics, 2018, doi: 10.25195/2017/4424.
[4] M. P. Ambara, M. Sudarma, and I. N. S. Kumara, “Desain Sistem Semantic Data Warehouse dengan Metode Ontology dan Rule Based untuk Mengolah Data Akademik Universitas XYZ di Bali,” Majalah Ilmiah Teknologi Elektro 2016, doi: 10.24843/mite.2016.v15i01p02.
[5] I. M. D. J. Sulastra, M. Sudarma, and I. N. S. Kumara, “PEMODELAN INTEGRASI NEARLY REAL TIME DATA WAREHOUSE DENGAN SERVICE ORIENTED ARCHITECTURE UNTUK MENUNJANG SISTEM INFORMASI RETAIL,” Majalah Ilmiah Teknologi Elektro 2015, doi: 10.24843/mite.2015.v14i02p03.
[6] M. R. Pasha, “Data Warehousing and the Unstructured Data,” Bahria Univercity Islam. Campus Gradute Resuslt, vol. DOI:10.1, 2016.
[7] S. Bouaziz, A. Nabli, and F. Gargouri, “From traditional data warehouse to real time data warehouse,” in Advances in Intelligent Systems and Computing, 2017, doi: 10.1007/978-3-319-53480-0_46.
[8] S. Bouaziz, A. Nabli, and F. Gargouri, “Design a data warehouse schema from document-oriented database,” in Procedia Computer Science, 2019, doi: 10.1016/j.procs.2019.09.177.
[9] F. Z. Al Faris, Suharjito, Diana, and A. Nugroho, “Development of Data Warehouse to Improve Services in IT Services Company,” in Proceedings of 2018 International Conference on Information Management and Technology, ICIMTech 2018, 2018, doi: 10.1109/ICIMTech.2018.8528146.
[10] H. Chandra, “Analysis of Change Data Capture Method in Heterogeneous Data Sources to Support RTDW,” in 2018 4th International Conference on Computer and Information Sciences: Revolutionising Digital Landscape for Sustainable Smart Society, ICCOINS 2018 - Proceedings, 2018, doi: 10.1109/ICCOINS.2018.8510574.
[11] I. G. Adnyana, M. Sudarma, and W. G. Ariastina, “Middleware ETL with CDC based on Event Driven Programming,” International Journal Of Engineering And Emerging Technology, vol. Vol. 3, No, 2018.
[12] A. Wibowo, “Problems and available solutions on the stage of Extract, Transform, and Loading in near real-time data warehousing (a literature study),” in 2015 International Seminar on Intelligent Technology and Its Applications, ISITIA 2015 - Proceeding, 2015, doi: 10.1109/ISITIA.2015.7220004.
[13] A. Sabtu et al., “The challenges of Extract, Transform and Loading (ETL) system implementation for near real-time environment,” in International Conference on Research and Innovation in Information Systems, ICRIIS, 2017, doi: 10.1109/ICRIIS.2017.8002467.
[14] R. P. Deb Nath, K. Hose, T. B. Pedersen, and O. Romero, “SETL: A programmable semantic extract-transform-load framework for semantic data warehouses,” Information Systems, 2017, doi: 10.1016/j.is.2017.01.005.
[15] N. Biswas, A. Sarkar, and K. C. Mondal, “Efficient incremental loading in ETL processing for real-time data integration,” Innovation in System Software Engineering, 2019, doi: 10.1007/s11334-019-00344-4.
[16] S. Thulasiram and N. Ramaiah, “Real Time Data Warehouse Updates Through Extraction-Transformation-Loading Process Using Change Data Capture Method,” 2020.
[17] B. Pan, G. Zhang, and X. Qin, “Design and realization of an ETL method in business intelligence project,” in 2018 3rd IEEE International Conference on Cloud Computing and Big Data Analysis, ICCCBDA 2018, 2018, doi: 10.1109/ICCCBDA.2018.8386526.
[18] Denny, I. P. M. Atmaja, A. Saptawijaya, and S. Aminah, “Implementation of change data capture in ETL process for data warehouse using HDFS and apache spark,” in Proceedings - WBIS 2017: 2017 International Workshop on Big Data and Information Security, 2018, doi: 10.1109/IWBIS.2017.8275102.
[19] I. Mekterović and L. Brkić, “Delta view generation for incremental loading of large dimensions in a data warehouse,” in 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics, MIPRO 2015 - Proceedings, 2015, doi: 10.1109/MIPRO.2015.7160496.
The Authors submitting a manuscript do so on the understanding that if accepted for publication, the copyright of the article shall be assigned to Jurnal Lontar Komputer as the publisher of the journal. Copyright encompasses exclusive rights to reproduce and deliver the article in all forms and media, as well as translations. The reproduction of any part of this journal (printed or online) will be allowed only with written permission from Jurnal Lontar Komputer. The Editorial Board of Jurnal Lontar Komputer makes every effort to ensure that no wrong or misleading data, opinions, or statements be published in the journal.
This work is licensed under a Creative Commons Attribution 4.0 International License.
