Abstract
The concept of memory disaggregation has recently been gaining traction in research. With memory disaggregation, data center compute nodes can directly access memory on adjacent nodes and are therefore able to overcome local memory restrictions, introducing a new data management paradigm for distributed computing. This paper proposes and demonstrates a memory disaggregated in-memory object store framework for big data applications by leveraging the newly introduced Thymes-isFlow memory disaggregation system. The framework extends the functionality of the pre-existing Apache Arrow Plasma object store framework to distributed systems by enabling clients to easily and efficiently produce and consume data objects across multiple compute nodes. This allows big data applications to increasingly leverage parallel processing at reduced development costs. In addition, the paper includes latency and throughput measurements that indicate only a modest performance penalty is incurred for remote disaggregated memory access as opposed to local (~6.5 vs ~5.75 GiB/s). The results can be used to guide the design of future systems that leverage memory disaggregation as well as the newly presented framework. This work is open-source and publicly accessible at https://doi.org/10.5281/zenodo.6368998.
Original language | English |
---|---|
Title of host publication | Proceedings of the 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) |
Editors | L. O'Conner |
Place of Publication | Piscataway |
Publisher | IEEE |
Pages | 1228-1234 |
Number of pages | 7 |
ISBN (Electronic) | 978-1-6654-9747-3 |
ISBN (Print) | 978-1-6654-9748-0 |
DOIs | |
Publication status | Published - 2022 |
Event | 2022 IEEE 36th International Parallel and Distributed Processing Symposium Workshops - Lyon, France Duration: 30 May 2022 → 3 Jun 2022 Conference number: 36th |
Publication series
Name | Proceedings - 2022 IEEE 36th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2022 |
---|
Conference
Conference | 2022 IEEE 36th International Parallel and Distributed Processing Symposium Workshops |
---|---|
Abbreviated title | IPDPSW 2022 |
Country/Territory | France |
City | Lyon |
Period | 30/05/22 → 3/06/22 |
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-careOtherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Keywords
- Memory Disaggregation
- Apache Arrow Plasma
- ThymesisFlow