Benchmarking Apache Arrow Flight - A wire-speed protocol for data transfer, querying and microservices

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

53 Downloads (Pure)

Abstract

Moving structured data between different big data frameworks and/or data warehouses/storage systems often cause significant overhead. Most of the time more than 80% of the total time spent in accessing data is elapsed in serialization/de-serialization step. Columnar data formats are gaining popularity in both analytics and transactional databases. Apache Arrow, a unified columnar in-memory data format promises to provide efficient data storage, access, manipulation and transport. In addition, with the introduction of the Arrow Flight communication capabilities, which is built on top of gRPC, Arrow enables high performance data transfer over TCP networks. Arrow Flight allows parallel Arrow RecordBatch transfer over networks in a platform and language-independent way, and offers high performance, parallelism and security based on open-source standards. In this paper, we bring together some recently implemented use cases of Arrow Flight with their benchmarking results. These use cases include bulk Arrow data transfer, querying subsystems and Flight as a microservice integration into different frameworks to show the throughput and scalability results of this protocol. We show that Flight is able to achieve up to 6000 MB/s and 4800 MB/s throughput for DoGet() and DoPut() operations respectively. On Mellanox ConnectX-3 or Connect-IB interconnect nodes Flight can utilize upto 95% of the total available bandwidth. Flight is scalable and can use upto half of the available system cores efficiently for a bidirectional communication. For query systems like Dremio, Flight is order of magnitude faster than ODBC and turbodbc protocols. Arrow Flight based implementation on Dremio performs 20x and 30x better as compared to turbodbc and ODBC connections respectively. We briefly outline some recent Flight based use cases both in big data frameworks like Apache Spark and Dask and remote Arrow data processing tools. We also discuss some limitations and future outlook of Apache Arrow and Arrow Flight as a whole.

Original languageEnglish
Title of host publicationProceedings of the 4th Benchmarking in the Datacenter: Expanding to the Cloud 2022
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Number of pages10
ISBN (Print)978-1-4503-9324-9
DOIs
Publication statusPublished - 2022
Event4th Benchmarking in the Datacenter: Expanding to the Cloud, BID 2022 - Seoul, Korea, Republic of
Duration: 2 Apr 20222 Apr 2022

Conference

Conference4th Benchmarking in the Datacenter: Expanding to the Cloud, BID 2022
Country/TerritoryKorea, Republic of
CitySeoul
Period2/04/222/04/22

Keywords

  • Apache arrow
  • Arrow flight
  • Data analytics
  • Data transfer
  • Databases
  • Microservices
  • Networks
  • Wire-protocol

Fingerprint

Dive into the research topics of 'Benchmarking Apache Arrow Flight - A wire-speed protocol for data transfer, querying and microservices'. Together they form a unique fingerprint.

Cite this