Scalable implementation of dependence clustering in Apache Spark
Ivannikova, E. (2017). Scalable implementation of dependence clustering in Apache Spark. In I. Škrjanc, & S. Blažič (Eds.), EAIS 2017 : Proceedings of the 2017 Evolving and Adaptive Intelligent Systems (EAIS) (pp. 1-6). IEEE. https://doi.org/10.1109/EAIS.2017.7954843
Date
2017Copyright
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
This article proposes a scalable version of the
Dependence Clustering algorithm which belongs to the class
of spectral clustering methods. The method is implemented
in Apache Spark using GraphX API primitives. Moreover, a
fast approximate diffusion procedure that enables algorithms
of spectral clustering type in Spark environment is introduced.
In addition, the proposed algorithm is benchmarked against
Spectral clustering. Results of applying the method to real-life
data allow concluding that the implementation scales well, yet
demonstrating good performance for densely connected graphs.
Publisher
IEEEParent publication ISBN
978-1-5090-6444-1Conference
Evolving and Adaptive Intelligent SystemsIs part of publication
EAIS 2017 : Proceedings of the 2017 Evolving and Adaptive Intelligent Systems (EAIS)ISSN Search the Publication Forum
2473-4691Keywords
Publication in research information system
https://converis.jyu.fi/converis/portal/detail/Publication/27339946