Pengolahan Big Data Dengan Sharding Database Dan Kappa Architecture Untuk Data Time-Series

Abstract

In the digital era, managing and processing Big Data presents challenges. Historical or time-series data is crucial for decision-making. Distributed database systems, specifically database sharding, efficiently distribute CPU load and memory usage. Kappa Architecture outperforms Lambda Architecture by 220% in terms of speed, though Lambda has slightly higher reliability. This research integrates database sharding with Kappa Architecture using Kaggle’s time-series data. Load testing showed 60.46% performance for data retrieval across distributed databases. Remarkably, the system’s reliability reached 100%, even when one of the services failed, handling 10000 new data entries. Furthermore, with the same configuration, it was found that the utilization of Kappa Architecture and sharding database resulted in a 70.26% better performance compared to the system solely implementing sharding database.