SymFormer: End-to-End Symbolic Regression Using Transformer-Based Architecture

Martin Vastl, Jonas Kulhanek, Jiri Kubalik, Erik Derner*, Robert Babuska

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

75 Downloads (Pure)

Abstract

Many real-world systems can be naturally described by mathematical formulas. The task of automatically constructing formulas to fit observed data is called symbolic regression. Evolutionary methods such as genetic programming have been commonly used to solve symbolic regression tasks, but they have significant drawbacks, such as high computational complexity. Recently, neural networks have been applied to symbolic regression, among which the transformer-based methods seem to be most promising. After training a transformer on a large number of formulas, the actual inference, i.e., finding a formula for new, unseen data, is very fast (in the order of seconds). This is considerably faster than state-of-the-art evolutionary methods. The main drawback of transformers is that they generate formulas without numerical constants, which have to be optimized separately, yielding suboptimal results. We propose a transformer-based approach called SymFormer, which predicts the formula by outputting the symbols and the constants simultaneously. This helps to generate formulas that fit the data more accurately. In addition, the constants provided by SymFormer serve as a good starting point for subsequent tuning via gradient descent to further improve the model accuracy. We show on several benchmarks that SymFormer outperforms state-of-the-art methods while having faster inference.

Original languageEnglish
Pages (from-to)37840-37849
Number of pages10
JournalIEEE Access
Volume12
DOIs
Publication statusPublished - 2024

Keywords

  • neural networks
  • Symbolic regression
  • transformers

Fingerprint

Dive into the research topics of 'SymFormer: End-to-End Symbolic Regression Using Transformer-Based Architecture'. Together they form a unique fingerprint.

Cite this