Relaxed functional dependency discovery in heterogeneous data lakes

Rihan Hai, Christoph Quix, Dan Wang

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

Functional dependencies are important for the definition of constraints and relationships that have to be satisfied by every database instance. Relaxed functional dependencies (RFDs) can be used for data exploration and profiling in datasets with lower data quality. In this work, we present an approach for RFD discovery in heterogeneous data lakes. More specifically, the goal of this work is to find RFDs from structured, semi-structured, and graph data. Our solution brings novelty to this problem in the following aspects: (1) We introduce a generic metamodel to the problem of RFD discovery, which allows us to define and detect RFDs for data stored in heterogeneous sources in an integrated manner. (2) We apply clustering techniques during RFD discovery for partitioning and pruning. (3) We performed an intensive evaluation with nine datasets, which shows that our approach is effective for discovering meaningful RFDs, reducing redundancy, and detecting inconsistent data.
Original languageEnglish
Title of host publicationInternational Conference on Conceptual Modeling
Pages225-239
Number of pages15
ISBN (Electronic)978-3-030-33223-5
DOIs
Publication statusPublished - 2019
Externally publishedYes

Fingerprint

Dive into the research topics of 'Relaxed functional dependency discovery in heterogeneous data lakes'. Together they form a unique fingerprint.

Cite this