Fuzzy Join for Flexible Combining Big Data Lakes in Cyber-Physical Systems

Cyber-physical systems produce large amounts of data that are stored in domain-related data lakes in a variety of formats. By using the big data technologies that enable efficient data processing, the value of the data increases, as these technologies can turn the data into actionable information that influences important decision-making processes. However, a broader view of the operational environment, an investigated phenomena, and challenges related to them can frequently be obtained after combining data from many data sets located in various big data lakes. This requires contact points in both data lakes that must be flexibly joined because in many cases, data sets do not correspond to one another directly. In this paper, we show fuzzy join operation for flexible combining big data lakes. The fuzzy join transforms numerical values of common attributes of joined data sets into fuzzy sets and uses such a representation in the join operation. We propose two variants of the join operation that transforms crisp numerical values of joining attributes into: 1) fuzzy numbers and 2) linguistic terms. The fuzzy join operation is implemented and tested in the declarative U-SQL language that is used for scalable and parallel querying in big data lakes. The ideas presented here are exemplified by a distributed analysis of cardiac disease data on Microsoft Azure cloud. The results of the conducted experiments confirm that the fuzzy join can enrich data sets that are used in making critical decisions and, as a highly scalable cloud-based solution, can be successfully used in processing large volumes of data delivered by cyber-physical systems.