An Integration Model on Data Lake for Solving Structured Data Silo Problems

Abstract

Data siloes are a major data management challenge in both public and business organizations. As a result of the organizational structure of work functions into numerous departments, each different department has distinct responsibilities and tends to create dependent software, applications and data systems to locally support its individual needs. Similar data are always stored in multiple silos or databases, and normally their data schema likes names and meanings are different from each other. As a result, users are perplexed on how to use those data coming from different silos of software applications. This research applies the data lake concept to solve the data silo problem. The scope of the research focuses on structured data silos. The objective of this research is to design a data lake architecture and its internal working framework by using Hive and Spark technologies to integrate data within a data lake and write functional testing programs in Java Spark. According to the result of testing based on a detailed developed framework, integrating data silos on data lakes can reduce data heterogeneity and data inconsistencies by 100%, and it was able to reduce the redundancy of the test data by 78.6% from the total of 13 separate data cases.