FPGA Acceleration of Zstd Compression Algorithm

Jianyu Chen, Maurice Daverveldt, Zaid Al-Ars

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

30 Downloads (Pure)

Abstract

With the continued increase in the amount of big data generated and stored in various application domains, such as high-frequency trading, compression techniques are becoming ever more important to reduce the requirements on communication bandwidth and storage capacity. Zstandard (Zstd) is emerging as an important compression algorithm for big data sets capable of achieving a good compression ratio but with a higher speed than comparable algorithms. In this paper, we introduce the architecture of a new hardware compression kernel for Zstd that allows the algorithm to be used for real-time compression of big data streams. In addition, we optimize the proposed architecture for the specific use case of streaming high-frequency trading data. The optimized kernel is implemented on a Xilinx Alveo U200 board. Our optimized implementation allows us to fit ten kernel blocks on one board, which is able to achieve a compression throughput of about 8.6GB/s and compression ratio of about 23.6%. The hardware implementation is open source and publicly available at https://github.com/ChenJianyunp/Hardware-Zstd-Compression-Unit.
Original languageEnglish
Title of host publication2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
EditorsL. O'Conner
Place of PublicationPiscataway
PublisherIEEE
Pages188-191
Number of pages4
ISBN (Electronic)978-1-6654-3577-2
ISBN (Print)978-1-6654-1192-9
DOIs
Publication statusPublished - 2021
Event2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) - Virtually at Portland, United States
Duration: 17 Jun 202121 Jun 2021

Workshop

Workshop2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
CountryUnited States
CityVirtually at Portland
Period17/06/2121/06/21

Bibliographical note

Accepted author manuscript

Keywords

  • FPGA
  • Zstd
  • compression

Fingerprint

Dive into the research topics of 'FPGA Acceleration of Zstd Compression Algorithm'. Together they form a unique fingerprint.

Cite this