Extending Unix Pipelines to DAGs

Research output: Contribution to journalArticleScientificpeer-review

11 Citations (Scopus)

Abstract

The Unix shell dgsh provides an expressive way to construct sophisticated and efficient non-linear pipelines. Such pipelines can use standard Unix tools, as well as third-party and custom-built components. Dgsh allows the specification of pipelines that perform non-uniform non-linear processing. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the processing task's throughput. A number of existing Unix tools have been adapted to take advantage of the new shell's multiple pipe input/output capabilities. The shell supports visualization of the process graphs, which can also aid debugging. Dgsh was evaluated through a number of common data processing and domain-specific examples, and was found to offer an expressive way to specify processing topologies, while also generally increasing processing throughput.

Original languageEnglish
Article number7903579
Pages (from-to)1547-1561
Number of pages15
JournalIEEE Transactions on Computers
Volume66
Issue number9
DOIs
Publication statusPublished - 1 Sept 2017
Externally publishedYes

Keywords

  • pipeline
  • pipes and filters architecture
  • Process-level parallelism
  • Unix

Fingerprint

Dive into the research topics of 'Extending Unix Pipelines to DAGs'. Together they form a unique fingerprint.

Cite this