Interpretable Scientific Discovery with Symbolic Regression: A Review
可解释科学发现与符号回归方法综述

Symbolic Regression (SR) is a rapidly growing subfield within machine learning (ML) to infer symbolic mathematical expressions from data [1, 2]. Interest in SR is being driven by the observation that it is not sufficient to only have accurate predictive models; however, it is often necessary that the learned models be interpretable [3]. A model is interpretable if the relationship between the input and output of the model can be logically or mathematically traced in a succinct manner. In other words, learnable models are interpretable if expressed as mathematical equations. As "disciplines" become increasingly data-rich and adopt ML techniques, the demand for interpretable models is likely to grow. For example, in the natural sciences (e.g., physics), mathematical models derived from first principles make it possible to reason about the underlying phenomenon in a way that is not possible with predictive models like deep neural networks. In critical disciplines like healthcare, non-interpretable models may never be allowed to be deployed - however accurate they maybe [4].
符号回归（Symbolic Regression, SR）是机器学习（ML）中一个快速发展的子领域，旨在从数据中推断符号化的数学表达式[1, 2]。对 SR 的兴趣源于以下观察：仅拥有高精度的预测模型是不够的，通常还需要所学模型具备可解释性[3]。若模型输入与输出之间的关系能以逻辑或数学方式简洁追溯，则该模型可被视为可解释。换言之，若以数学方程形式表达，可学习模型即具备可解释性。随着各"学科"数据日益丰富并采用 ML 技术，对可解释模型的需求可能持续增长。例如在自然科学（如物理学）中，基于第一性原理推导的数学模型能以深度神经网络等预测模型无法实现的方式解析底层现象。而在医疗保健等关键领域，无论非可解释模型精度多高，都可能永远无法获准部署[4]。

Example: Consider a data set consisting of samples $\left(q_1,q_2,r,F\right)$ ,where $q_1$ and $q_2$ are the charges of two particles, $r$ is the distance between them and $F$ is the measured force between the particles. Assume $q_1,q_2$ , and $r$ are the input variables,and $F$ is the output variable. Suppose we model the input-output relationship as $F={\theta }_0+{\theta }_1{q}_1+{\theta }_2{q}_2+{\theta }_{3}r$ . Then,using the data set,we can infer the model’s parameters $\left({\theta }_i\right)$ . The model will be interpretable because we will know the impact of each variable on the output. For example,if ${\theta }_3$ is negative,then that implies that as $r$ increases,the force $F$ will decrease. From physics,we know that this form of the model is unlikely to be accurate. On the other hand, we could model the relationship using a neural network $F=NN\left(q_1,q_2,r,\theta \right)$ . We expect the model to be highly accurate and predictive because neural networks (NNs) are universal function approximators. However, the model is uninterpretable because the input-output relationship is not easily apparent. The input feature vector subsequently undergoes several layers of nonlinear transformations,i.e., $y=\sigma \left(\sum _i{W}_i\sigma \left(\sum _j{W}_j\sigma \left(\sum _k{W}_k\sigma \left(\cdots \sum _{\ell }{W}_{\ell }\mathbf{x}\right)\right)\right)\right)$ ,where $\sigma$ is a nonlinear activation function,and $W_{idx}$ are the learnable parameters of NN layer of index idx. Such models,called "blackbox",do not have an internal logic to let users understand how inputs are mathematically mapped to outputs. Explainability is the application of other methods to explain model predictions and to understand how it is learned. It refers to why the model makes the decision that way. What distinguishes explainability from interpretability is that interpretable models are transparent [3]. For example, the linear regression model predictions can be interpreted by evaluating the relative contribution of individual features to the predictions using their weights. An ideal SR model will return the relationship as $k\frac{q_1{q}_2}{r^2}$ ,which is the definition of the Coulomb force between two charged particles with a constant $1k=8.98\times {10}^9$ . However,learning the SR model is highly non-trivial as it involves searching over a large space of mathematical operations and identifying the right constant(k)that will fit the data. SR models can be directly inferred from data or can be used to "whitebox" a "blackbox" model such as a neural network.
示例：考虑一个由样本 $\left(q_1,q_2,r,F\right)$ 组成的数据集，其中 $q_1$ 和 $q_2$ 是两个粒子的电荷， $r$ 是它们之间的距离， $F$ 是测量到的粒子间作用力。假设 $q_1,q_2$ 、 $r$ 为输入变量， $F$ 为输出变量。若将输入-输出关系建模为 $F={\theta }_0+{\theta }_1{q}_1+{\theta }_2{q}_2+{\theta }_{3}r$ ，则可通过数据集推断模型参数 $\left({\theta }_i\right)$ 。该模型具有可解释性，因为我们将了解每个变量对输出的影响。例如，若 ${\theta }_3$ 为负值，则意味着随着 $r$ 增大，作用力 $F$ 会减小。从物理学角度看，这种模型形式可能不够精确。另一方面，我们也可以使用神经网络 $F=NN\left(q_1,q_2,r,\theta \right)$ 来建模该关系。由于神经网络（NNs）是通用函数逼近器，我们预期该模型具有高度准确性和预测能力。然而，该模型不可解释，因为输入-输出关系不易直观呈现。输入特征向量随后会经历多层非线性变换，即 $y=\sigma \left(\sum _i{W}_i\sigma \left(\sum _j{W}_j\sigma \left(\sum _k{W}_k\sigma \left(\cdots \sum _{\ell }{W}_{\ell }\mathbf{x}\right)\right)\right)\right)$ ，其中 $\sigma$ 是非线性激活函数， $W_{idx}$ 是指数为 idx 的神经网络层可学习参数。 这类被称为“黑箱”的模型缺乏内部逻辑机制，无法让用户理解输入如何通过数学映射转化为输出。可解释性则是应用其他方法来阐明模型预测结果并理解其学习过程，核心在于揭示模型为何做出特定决策。与可解释性不同，可解释模型具有透明特性[3]——例如线性回归模型可通过特征权重评估各变量对预测结果的相对贡献度来实现解释。理想的符号回归(SR)模型会返回如 $k\frac{q_1{q}_2}{r^2}$ 所示的关系式，该式实为带常数项 $1k=8.98\times {10}^9$ 的两个带电粒子间库仑力的定义式。然而，SR 模型的学习过程极具挑战性，需在庞大数学运算符空间中进行搜索，并确定能拟合数据的最优常数(k)。SR 模型既可直接从数据中推导获得，也可用于将神经网络等"黑箱"模型转化为"白箱"模型。

The ultimate goal of SR is to bridge data and observations following the Keplerian trial and error approach [5]. Kepler developed a data-driven model for planetary motion using the most accurate astronomical measurements of the era, which resulted in elliptic orbits described by a power law. In contrast, Newton developed a dynamic relationship between physical variables that described the underlying process at the origin of these elliptic orbits. Newton's approach [6] led to three laws of motion later verified by experimental observations. Whereas both methods fit the data well, Newton's approach could be generalized to predict behavior in regimes where no data were available. Although SR is regarded as a data-driven model discovery tool, it aims to find a symbolic model that simultaneously fits data well and could be generalized to uncovered regimes.
SR 的终极目标是遵循开普勒的试错法[5]，在数据与观测之间架起桥梁。开普勒利用当时最精确的天文测量数据，为行星运动开发了一个数据驱动模型，最终得出由幂律描述的椭圆轨道。相比之下，牛顿则建立了物理变量之间的动力学关系，揭示了这些椭圆轨道背后的成因过程。牛顿的方法[6]后来被实验观测验证，形成了三大运动定律。尽管两种方法都能很好地拟合数据，但牛顿的方法能够推广到尚无数据的领域以预测行为。虽然 SR 被视为数据驱动的模型发现工具，但其目标是找到一个既能良好拟合数据、又能推广到未知领域的符号模型。

SR is deployed as an interpretable and predictive ML model or a data-driven scientific discovery method. SR was investigated as early as 1970 in research works [7, 8, 9] aiming to rediscover empirical laws. Such works iteratively apply a set of data-driven heuristics to formulate mathematical expressions. The first AI system meant to automate scientific discovery is called BACON [10, 11]. It was developed by Patrick Langley in the late 1970s and was successful in rediscovering versions of various physical laws, such as Coulomb's law and Galileo's laws for the pendulum and constant acceleration, among many others. SR was later studied by Koza [12, 13, 1] who proposed that genetic programming (GP) can be used to discover symbolic models by encoding mathematical expressions as computational trees, where GP is an evolutionary algorithm that iteratively evolves an initial population of individuals via biology-inspired operations. SR was since then tackled with GP-based methods [1, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]. Moreover, it was popularized as a data-driven scientific discovery tool with the commercial software Eureqa [26] based on a research work [2]. Whereas GP-based methods achieve high prediction accuracy, they do not scale to high dimensional data sets and are sensitive to hyperparameters [19]. More recently, SR has been addressed with deep learning-based methods 27. 28, 19, 29, 30, 31, 32, 33 which leverage neural networks (NNs) to learn accurate symbolic models. SR has been applied in fundamental and applied sciences such as astrophysics [34], chemistry [35, 36], materials science [37, 38], semantic similarity measurement [39], climatology 40, medicine [41], among many others. Many of these applications are promising, showing the potential of SR. A recent SR benchmarking platform SRBench is introduced by La Cava et al. [42]. It comprises 14 SR methods (among which ten are GP-based), applied on 252 data sets. The goal of SRBench was to provide a benchmark for rigorous evaluation and comparison of SR methods.
SR 作为一种可解释的预测性机器学习模型或数据驱动的科学发现方法被部署。早在 1970 年的研究工作中[7,8,9]，SR 就被探索用于重新发现经验定律。这类工作通过迭代应用一系列数据驱动的启发式方法来构建数学表达式。首个旨在自动化科学发现的人工智能系统名为 BACON[10,11]，由 Patrick Langley 于 1970 年代末开发，成功重新发现了包括库仑定律、伽利略摆锤定律和匀加速定律在内的多种物理定律版本。随后 Koza[12,13,1]研究了 SR，提出可通过将数学表达式编码为计算树，利用遗传编程(GP)发现符号模型——GP 是一种通过生物启发操作迭代进化初始个体群体的进化算法。此后 SR 主要通过基于 GP 的方法进行研究[1,14-25]。此外，基于研究工作[2]的商业软件 Eureqa[26]将其推广为数据驱动的科学发现工具。 基于高斯过程（GP）的方法虽然预测精度高，但无法扩展到高维数据集且对超参数敏感[19]。最近，符号回归（SR）开始采用基于深度学习的方法[27, 28, 19, 29, 30, 31, 32, 33]，这些方法利用神经网络（NNs）学习精确的符号模型。SR 已广泛应用于基础科学与应用科学领域，如天体物理学[34]、化学[35, 36]、材料科学[37, 38]、语义相似性度量[39]、气候学[40]、医学[41]等。许多应用展现了 SR 的巨大潜力。La Cava 等人[42]近期推出了 SR 基准测试平台 SRBench，该平台包含 14 种 SR 方法（其中 10 种基于 GP），应用于 252 个数据集，旨在为 SR 方法的严格评估与比较提供基准。

${}^{1}k$ is the electric force (or Coulomb) constant, $k=8.9875517923\times {10}^9\mathrm{kg}\cdot {\mathrm{m}}^3\cdot {\mathrm{s}}^{-4}\cdot {\mathrm{A}}^{-2}$ in SI base units.
${}^{1}k$ 是电力（或库仑）常数， $k=8.9875517923\times {10}^9\mathrm{kg}\cdot {\mathrm{m}}^3\cdot {\mathrm{s}}^{-4}\cdot {\mathrm{A}}^{-2}$ 以国际单位制基本单位表示。

This survey aims to help researchers effectively and comprehensively understand the SR problem and how it could be solved, as well as to present the current status of the advances made in this growing subfield. The survey is structured as follows. First, we define the SR problem, present a structured and comprehensive review of methods, and discuss their strengths and limitations. Furthermore, we discuss the adoption of these SR methods across various application domains and assess their effectiveness. Along with this survey, a living review [25] aims to group state-of-the-art SR methods and applications and track advances made in the SR field. The objective is to update this list often to incorporate new research works.
本综述旨在帮助研究者有效且全面地理解 SR 问题及其解决途径，并展示这一快速发展的子领域当前的研究进展。综述结构如下：首先，我们定义 SR 问题，对现有方法进行结构化、全面的梳理，并探讨其优势与局限。此外，我们分析了这些 SR 方法在不同应用领域的采用情况，评估其实际效果。配合本综述，动态文献[25]将持续汇总最前沿的 SR 方法与应用案例，追踪该领域进展，通过定期更新纳入最新研究成果。

This paper is organized as follows. The SR problem definition is presented in Section 2. We present an overview of methods deployed to solve the SR problem in Section 3, and the methods are discussed in detail in Section 4.5 and 6. Selected applications are described and discussed in Section 7. Section 8 presents an overview of existing benchmark data sets. Finally, we summarize our conclusions and discuss perspectives in Section 10.
本文组织结构如下：第 2 节阐述 SR 问题定义；第 3 节概述 SR 问题解决方法框架；第 4.5 节与第 6 节将详细解析具体方法；第 7 节选取典型应用场景进行论述；第 8 节汇总现有基准数据集；最后在第 10 节总结结论并展望未来研究方向。

The problem of symbolic regression can be defined in terms of classical Empirical Risk Minimization (ERM) [43].
符号回归问题可以依据经典的经验风险最小化（ERM）框架进行定义[43]。

Data: Given a data set $\mathcal{D}={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^n$ ,where ${\mathbf{x}}_i\in {\mathbb{R}}^d$ is the input vector and $y_i\in \mathbb{R}$ is a scalar output.
数据：给定数据集 $\mathcal{D}={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^n$ ，其中 ${\mathbf{x}}_i\in {\mathbb{R}}^d$ 为输入向量， $y_i\in \mathbb{R}$ 为标量输出。

Function Class: Let $\mathcal{F}$ be a function class consisting of mappings $f:{\mathbb{R}}^d\to \mathbb{R}$ .
函数类：设 $\mathcal{F}$ 为由映射 $f:{\mathbb{R}}^d\to \mathbb{R}$ 构成的函数类。

Loss Function: Define the loss function for every candidate $f\in \mathcal{F}$ :
损失函数：为每个候选函数 $f\in \mathcal{F}$ 定义损失函数：

A common choice is the squared difference between the output and prediction,i.e. $l\left(f\right)=\sum _i{(y_i-f\left(x_i\right))}^2$ .
常见的选择是输出与预测之间的平方差，即 $l\left(f\right)=\sum _i{(y_i-f\left(x_i\right))}^2$ 。

Optimization: The optimization task is to find the function(f)over the set of functions $\mathcal{F}$ that minimizes the loss function:
优化：优化任务是在函数集 $\mathcal{F}$ 上找到使损失函数最小化的函数(f)。

As stated below, what distinguishes SR from conventional regression problems is the discrete nature of the function class $\mathcal{F}$ . Different methods for solving the SR problem reduce to characterizing the function class.
如下所述，符号回归与传统回归问题的区别在于函数类 $\mathcal{F}$ 的离散性质。解决符号回归问题的不同方法归结为对函数类的刻画。

In SR,to define $\mathcal{F}$ ,we specify a library of elementary arithmetic operations and mathematical functions and variables,and an element $f\in \mathcal{F}$ is the set of all functions that can be obtained by function composition in the library [44]. For example, consider a library:
在符号回归（SR）中，为了定义 $\mathcal{F}$ ，我们指定了一个包含基本算术运算、数学函数及变量的库，而 $f\in \mathcal{F}$ 则是通过该库中函数组合所能获得的所有函数的集合[44]。例如，考虑如下库：

Then the set of all of the polynomials (in one variable $x$ ) with integer coefficients can be derived from $L$ using function composition.
那么所有具有整数系数的（单变量 $x$ ）多项式集合都可以通过函数复合从 $L$ 导出。

It is convenient to express symbolic expressions in a sequential form using either a unary-binary expression tree or the polish notation [45]. For example,the expression $f\left(\mathrm{x}\right)=x_1{x}_2-2x_3$ can be derived using function composition from $L$ (Eq. 3) and represented as a tree-like structure illustrated in Figure 1a. By traversing the (binary) tree top to bottom and left to right in a depth-first manner, we can represent the same expression as a unique sequence called the polish form, as illustrated in Figure 1b.
为便于表达符号式，可采用一元二元表达式树或波兰表示法[45]将其转换为序列形式。例如，表达式 $f\left(\mathrm{x}\right)=x_1{x}_2-2x_3$ 可通过函数组合从 $L$ （式 3）派生而来，并以图 1a 所示的树状结构呈现。通过自上而下、从左至右深度优先遍历该（二叉）树，可将同一表达式表示为唯一的序列形式，称为波兰式，如图 1b 所示。

Figure 1: (a) Example of a unary-binary tree that encodes $f\left(\mathrm{x}\right)=x_1{x}_2-2x_3$ . (b) Sequence representation of the tree-like structure of $f\left(\mathrm{x}\right)$ .
图 1：(a) 编码 $f\left(\mathrm{x}\right)=x_1{x}_2-2x_3$ 的一元二元树示例。(b) $f\left(\mathrm{x}\right)$ 树状结构的序列表示形式。

In practice,the library $L$ includes many other common elementary mathematical functions,including the basic trigonometric functions like sine, cosine, logarithm, exponential, square root, power low, etc. Prior domain knowledge is advantageous for library definition because it reduces the search space to only include the most relevant mathematical operations to the studied problem. Furthermore, a large range of possible numeric constants should be possible to express. For example, numbers in base-10 floating point notation rounded up to four significant digits can be represented as triple of (sign, mantissa, exponent) [31]. The function sin(3.456x), for example,can be represented as $\left[\sin ,\text{ mul,}3456,E-3,x\right]$ .
实际上，该库 $L$ 包含了许多其他常见的初等数学函数，包括基本三角函数如正弦、余弦、对数、指数、平方根、幂律等。先验领域知识对于库的定义是有利的，因为它将搜索空间缩小到仅包含与研究问题最相关的数学运算。此外，应能表达大范围的可能的数值常量。例如，以十进制浮点表示法四舍五入至四位有效数字的数字可以表示为（符号、尾数、指数）的三元组[31]。例如，函数 sin(3.456x)可以表示为 $\left[\sin ,\text{ mul,}3456,E-3,x\right]$ 。

In this survey, we categorize SR methods in the following manner: regression-based methods, expression tree-based methods, physics-inspired and mathematics-inspired methods, as presented in Figure 2. For each category, a summary of the mathematical tool, the expression form, the set of unknowns, and the search space, is presented in Table 1.
在本综述中，我们将符号回归方法分类如下：基于回归的方法、基于表达式树的方法、物理启发和数学启发的方法，如图 2 所示。每个类别的数学工具、表达式形式、未知数集合和搜索空间的总结见表 1。

The linear method defines the functional form as a linear combination of nonlinear functions of $x$ that are comprised in the predefined library $L$ . Linear models are expressed as:
线性方法将函数形式定义为由预定义库 $L$ 中包含的 $x$ 非线性函数的线性组合。线性模型表示为：

where $j$ spans the base functions of $L$ . The optimization problem reduces to find the set of parameters $\{\theta \}$ that minimizes the loss function defined over a continuous parameter space $\mathrm{\Theta }={\mathbb{R}}^M$ as follows:
其中 $j$ 遍历 $L$ 的基函数。优化问题简化为寻找参数集 $\{\theta \}$ ，以最小化在连续参数空间 $\mathrm{\Theta }={\mathbb{R}}^M$ 上定义的损失函数，如下所示：

This method is advantageous for being deterministic and disadvantageous because it imposes a single model structure which is fixed during training when the model's parameters are learned.
该方法的优势在于其确定性，劣势则在于它强制采用单一模型结构，在训练学习模型参数时该结构保持固定不变。

The nonlinear method defines the model structure by a neural network. Nonlinear models can thus be expressed as:
非线性方法通过神经网络定义模型结构。因此，非线性模型可表示为：

where $\sigma$ is a nonlinear activation function,and $W_{idx}$ are the learnable parameters of NN layer of index $idx$ . Similarly to the linear method,the optimization problem reduces finding the set of parameters $\{W,b\}$ of neural network layers, which minimizes the loss function over the space of real values.
其中 $\sigma$ 是一个非线性激活函数， $W_{idx}$ 是索引为 $idx$ 的神经网络层的可学习参数。与线性方法类似，优化问题简化为寻找神经网络层的参数集 $\{W,b\}$ ，该参数集在实数空间上最小化损失函数。

Expression tree-based methods treat mathematical expressions as unary-binary trees whose internal nodes are operators and terminals are operands (variables or constants). This category comprises GP-based, deep neural transformers, and reinforcement learning-based methods. In GP-based methods, a set of transition rules (e.g., mutation, crossover) is defined over the tree space and applied to an initial population of trees throughout many iterations until the loss function is minimized. Transformers [46] represent a novel architecture of neural network (encoder and decoder) that uses attention mechanism. The latter was primarily used to capture long-range dependencies in a sentence. Transformers were designed to operate on sequential data and to perform sequence-to-sequence (seq2seq) tasks. For their use in SR,input data points(x,y)and symbolic expressions (f)are encoded as sequences and transformers perform set-to-sequence tasks. The unknowns are the weight parameters of the encoder and the decoder. Reinforcement learning (RL) is a machine learning method that seeks to learn a policy $\pi \left(x\mid \theta \right)$ by training an agent to perform a task by interacting with its environment in discrete time steps. An RL setting requires four components: state space, action space, state transition probabilities, and reward. The agent selects an action that is sent to the environment. A reward and a new state are sent back to the agent from its environment and used by the agent to improve its policy at the next time step. In the context of SR, symbolic expression (sequence) represents a state, predicting an element in a sequence represents an action, the parent and sibling represent the environment, and the reward is commonly chosen as the mean square error (MSE). RL-based SR methods are commonly hybrid and use various ML tools (e.g., NN, RNN, etc.) in a joint manner with RL.
基于表达式树的方法将数学表达式视为一元-二元树，其内部节点为运算符，叶节点为操作数（变量或常量）。这类方法包括基于遗传编程（GP）、深度神经变换器和强化学习的方法。在基于 GP 的方法中，定义了一组在树空间上的转换规则（如变异、交叉），并在多次迭代中应用于初始树种群，直到损失函数最小化。变换器[46]代表了一种新颖的神经网络架构（编码器和解码器），它使用注意力机制。后者最初用于捕捉句子中的长距离依赖关系。变换器设计用于处理序列数据并执行序列到序列（seq2seq）任务。在符号回归（SR）中的应用中，输入数据点(x,y)和符号表达式(f)被编码为序列，变换器执行集合到序列的任务。未知的是编码器和解码器的权重参数。 强化学习（RL）是一种机器学习方法，旨在通过训练智能体在离散时间步中与环境交互来完成任务，从而学习策略 $\pi \left(x\mid \theta \right)$ 。RL 设定需要四个组成部分：状态空间、动作空间、状态转移概率和奖励。智能体选择一个动作发送给环境，环境则返回奖励和新状态给智能体，用于在下一时间步改进其策略。在符号回归（SR）的上下文中，符号表达式（序列）代表状态，预测序列中的元素代表动作，父节点和兄弟节点代表环境，而奖励通常选择均方误差（MSE）。基于 RL 的 SR 方法通常是混合型的，并联合使用多种机器学习工具（如神经网络 NN、循环神经网络 RNN 等）与 RL 协同工作。

Figure 2: Taxonomy based on the type of symbolic regression methods. $\varphi$ denotes a neural network function, $W$ denotes the set of learnable parameters in NN. $\mathbf{x}$ denotes the input data, $\mathbf{z}$ denotes a reduced representation of $\mathbf{x}$ ,and ${\mathbf{x}}^{\prime }$ denotes a new representation of $\mathbf{x}$ ,e.g.,by defining new features based on the original ones. $\mathcal{T}$ represents the final population of selected expression trees in genetic programming.
图 2：基于符号回归方法类型的分类体系。 $\varphi$ 表示神经网络函数， $W$ 表示神经网络中的可学习参数集合， $\mathbf{x}$ 表示输入数据， $\mathbf{z}$ 表示 $\mathbf{x}$ 的简化表示， ${\mathbf{x}}^{\prime }$ 表示 $\mathbf{x}$ 的新表示（例如通过基于原始特征定义新特征）， $\mathcal{T}$ 代表遗传编程中最终选定的表达式树种群。

The linear approach assumes,by definition,that the target symbolic expression $\left(f\left(x\right)\right)$ is a linear combination of nonlinear functions of feature attributes:
线性方法默认假设目标符号表达式 $\left(f\left(x\right)\right)$ 是特征属性非线性函数的线性组合：

Here $\mathrm{x}$ denotes the input features vector, ${\theta }_j$ denotes a weight coefficient,and $h_j(\cdot )$ denotes a unary operator of the library $L$ . This approach predefines the model’s structure and reduces the SR problem to learn only the model’s parameters by solving a system of linear equations. The particular case where $f\left(x\right)$ is a linear combination of degree-one monomial reduces to a conventional linear regression problem,i.e., $f\left(x\right)=\sum _j{\theta }_j{x}^j=$ ${\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+\cdots$ . There exist two cases for this problem: (1) a unidimensional case defined by $f:{\mathbb{R}}^d\to \mathbb{R}$ ; and (2) a multidimensional case defined by $f:{\mathbb{R}}^d\to {\mathbb{R}}^m$ ,with $d$ the number of input features and $m$ the number of variables required for a complete description of a system; for example, the Lorenz system for fluid flow is defined in terms of three physical variables which depend on time.
此处 $\mathrm{x}$ 表示输入特征向量， ${\theta }_j$ 表示权重系数， $h_j(\cdot )$ 表示库 $L$ 中的一元运算符。该方法预定义了模型结构，并将符号回归问题简化为仅通过求解线性方程组学习模型参数。当 $f\left(x\right)$ 为一次单项式的线性组合时，特例退化为常规线性回归问题，即 $f\left(x\right)=\sum _j{\theta }_j{x}^j=$ ${\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+\cdots$ 。该问题存在两种情况：(1) 由 $f:{\mathbb{R}}^d\to \mathbb{R}$ 定义的单维情形；(2) 由 $f:{\mathbb{R}}^d\to {\mathbb{R}}^m$ 定义的多维情形，其中 $d$ 表示输入特征数量， $m$ 表示完整描述系统所需的变量数；例如，描述流体流动的洛伦兹系统就以三个随时间变化的物理变量定义。

Table 1: Table summarizing symbolic regression methods. The mathematical tool, the expression form, the set of unknowns, and the search space are specified for each method. Set2seq abbreviates "set-to-sequence".
表 1：符号回归方法汇总表。每种方法均标注了数学工具、表达式形式、未知数集合及搜索空间。Set2seq 为"集合到序列"的缩写。

Given a data set $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^n$ ,the mathematical expression could be either univariate $\left(x_i\in \mathbb{R},y_i=f\left(x_i\right)\right)$ or multivariate $\left({\mathbf{x}}_i\in {\mathbb{R}}^d,y_i=f\left({\mathbf{x}}_i\right)\right)$ . The methodology of linear SR is presented in detail for the univariate case in Secion 4.1.1 for simplicity and is extended for the multivariate case in Section 4.1.2
给定数据集 $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^n$ ，数学表达式可以是单变量 $\left(x_i\in \mathbb{R},y_i=f\left(x_i\right)\right)$ 或多变量 $\left({\mathbf{x}}_i\in {\mathbb{R}}^d,y_i=f\left({\mathbf{x}}_i\right)\right)$ 。为简化起见，4.1.1 节详细介绍了单变量情况下的线性符号回归方法，并在 4.1.2 节中扩展至多变量情形。

Data set: $\mathcal{D}=\left\{x_i\in \mathbb{R};y_i=f\left(x_i\right)\right\}$ .  数据集： $\mathcal{D}=\left\{x_i\in \mathbb{R};y_i=f\left(x_i\right)\right\}$ 。

Library: $L$ can include any number of mathematical operators such that the dimension of the data set is always greater than the dimension of the library matrix (see discussion below).
库： $L$ 可包含任意数量的数学运算符，确保数据集的维度始终大于库矩阵的维度（详见下文讨论）。

In this approach,a coefficient ${\theta }_j$ is assigned to each candidate function $\left(f_j(\cdot )\in L\right)$ as an activeness criterion such that:
在此方法中，为每个候选函数 $\left(f_j(\cdot )\in L\right)$ 分配一个系数 ${\theta }_j$ 作为活跃性准则，具体表现为：

Applying Eq. 8 to input-output pairs $\left(x_i,y_i\right)$ yields a system of linear equations as follows:
将方程 8 应用于输入-输出对 $\left(x_i,y_i\right)$ ，可得到如下线性方程组：

which can be represented in a matrix form as:
这可以用矩阵形式表示为：

Equation 10 can then be presented in a compact form:
随后，方程 10 可以简洁地表示为：

where $\mathrm{\Theta }\in {\mathbb{R}}^{(k+1)}$ is the sparse vector of coefficients,and $\mathrm{U}\in {\mathbb{R}}^{n\times \left(k+1\right)}$ is the library matrix which can be represented as a function of the input vector $\mathrm{X}$ as follows:
其中 $\mathrm{\Theta }\in {\mathbb{R}}^{(k+1)}$ 为稀疏系数向量， $\mathrm{U}\in {\mathbb{R}}^{n\times \left(k+1\right)}$ 是库矩阵，可表示为输入向量 $\mathrm{X}$ 的函数，具体如下：

Example: For a library defined as:
示例：对于一个定义为以下的库：

The matrix $\mathrm{U}$ becomes:  矩阵 $\mathrm{U}$ 变为：

Each row (of index $i$ ) in Eq. 12 is a vector of $\left(k+1\right)$ functions of $x_i$ . The vector of coefficients,i.e.,the model’s parameters, is obtained by solving Eq. 11 as follows ${}^2$ :
方程 12 中的每一行（索引 $i$ ）是一个由 $\left(k+1\right)$ 个关于 $x_i$ 的函数组成的向量。系数向量（即模型参数）通过求解方程 11 获得，具体如下 ${}^2$ ：

The magnitude of a coefficient ${\theta }_k$ effectively measures the size of the contribution of the associated function $f_k(\cdot )$ to the final prediction. Finally,the prediction vector $\hat{\mathrm{Y}}$ can be evaluated using Eq. 11,
系数 ${\theta }_k$ 的幅值实质上衡量了关联函数 $f_k(\cdot )$ 对最终预测的贡献大小。最终，预测向量 $\hat{\mathrm{Y}}$ 可通过方程 11 计算得出，

An exemplary schematic is illustrated in Figure 3 for the univariate function $f\left(x\right)=1+\alpha x^3$ . Only coefficients associated with functions $\left\{1,x^3\right\}$ of the library are non-zero,with values equal to 1 and $\alpha$ ,respectively.
图 3 展示了一个单变量函数 $f\left(x\right)=1+\alpha x^3$ 的示例示意图。仅与函数库中 $\left\{1,x^3\right\}$ 相关联的系数为非零值，其数值分别为 1 和 $\alpha$ 。

Figure 3: Schematic of the system of linear equations of Eq. 11 for $f\left(x\right)=1+\alpha x^3$ . A library matrix $\mathrm{U}\left(\mathrm{X}\right)$ of nonlinear functions of the input is constructed,where $L=\left\{1,x,x^2,x^3,\cdots \right\}$ . The marked entries in the $\mathrm{\Theta }$ vector denote the non-zero coefficients determining which functions of the library are active.
图 3：针对 $f\left(x\right)=1+\alpha x^3$ 的方程 11 线性方程组示意图。构建了一个输入非线性函数的库矩阵 $\mathrm{U}\left(\mathrm{X}\right)$ ，其中 $L=\left\{1,x,x^2,x^3,\cdots \right\}$ 。 $\mathrm{\Theta }$ 向量中标记的条目表示决定库中哪些函数被激活的非零系数。

In the following, linear SR is tested on synthetic data. In each experiment, training and test data sets are generated. Each set consists of twenty data points randomly sampled from a uniform distribution $\mathrm{U}\left(-1,1\right)$ , and $y$ is evaluated using a univariate function,i.e., $\mathcal{D}={\left\{(x_i,f\left(x_i\right))\right\}}_{i=1}^n$ . Two libraries are considered in these experiments: $L_1=\left\{x,{(\cdot )}^2,{(\cdot )}^3,\cdots ,{(\cdot )}^9\right\}$ and $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ . The results are reported in terms of the output expression (Equation 7) and the coefficient of determination $R^2$ . SR problems are grouped into (i) pure polynomial functions and (ii) mixed polynomial and trigonometric functions. In each experiment, parameters are learned using the training data set, and results are reported for the test data set in
在以下实验中，线性符号回归（SR）在合成数据上进行了测试。每组实验都会生成训练数据集和测试数据集。每个数据集包含从均匀分布 $\mathrm{U}\left(-1,1\right)$ 中随机采样的二十个数据点，并使用单变量函数（即 $\mathcal{D}={\left\{(x_i,f\left(x_i\right))\right\}}_{i=1}^n$ ）对 $y$ 进行评估。实验中考虑了两个库： $L_1=\left\{x,{(\cdot )}^2,{(\cdot )}^3,\cdots ,{(\cdot )}^9\right\}$ 和 $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ 。结果以输出表达式（公式 7）和决定系数 $R^2$ 的形式呈现。符号回归问题分为两类：(i)纯多项式函数和(ii)混合多项式与三角函数。每组实验中，参数通过训练数据集学习，并在测试数据集上报告结果。

For polynomial functions,an exact output is obtained using $L_1$ with an $R^2=1.0$ ,whereas only approximate output is obtained using $L_2$ . In the latter case,the quality of the fit depends on the size of the training data set. An exemplary result is shown in Figure 4 for $f\left(x\right)=x+x^2+x^3$ . Points represent the (test) data of the input file,i.e.,X; the red curve represents $f\left(x\right)$ as a function of $x$ ,and the blue and black dashed curves represent the predicted function $\hat{f}\left(x\right)$ obtained using $L_1$ and $L_2$ respectively. An exact match between the ground-truth function and the predicted one is found using $L_1$ ,whereas a significant discrepancy is obtained using $L_2$ . This discrepancy could be explained by the fact that various functions in $L_2$ exhibit the same $x$ -dependence over the covered $x$ -range.
对于多项式函数，使用 $L_1$ 配合 $R^2=1.0$ 可获得精确输出，而使用 $L_2$ 仅能得到近似结果。后一种情况下，拟合质量取决于训练数据集的大小。图 4 展示了 $f\left(x\right)=x+x^2+x^3$ 的典型结果：点代表输入文件中的（测试）数据，即 X；红色曲线表示作为 $x$ 函数的 $f\left(x\right)$ ，蓝色和黑色虚线则分别代表通过 $L_1$ 和 $L_2$ 得到的预测函数 $\hat{f}\left(x\right)$ 。使用 $L_1$ 时，真实函数与预测函数完全吻合，而使用 $L_2$ 则存在显著差异。这种差异可解释为：在 $L_2$ 中，多种函数在覆盖的 $x$ 范围内表现出相同的 $x$ 依赖性。

${}^2$ Technically the pseudo-inverse, $U^{+}$
${}^2$ 技术上称为伪逆， $U^{+}$

Table 2: Results of linear SR in the case of univariate functions. $\mathcal{D}=\left\{\left(x_i;y_i\right)\right\};x_i\in \mathrm{U}\left(-1,1,20\right)$ and $y_i=f\left(x_i\right)$ . $L_1=\left\{x,{(\cdot )}^2,\cdots ,{(\cdot )}^9\right\}$ and $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ . T denotes True,and $\mathrm{F}$ denotes False.
表 2：单变量函数线性符号回归结果。 $\mathcal{D}=\left\{\left(x_i;y_i\right)\right\};x_i\in \mathrm{U}\left(-1,1,20\right)$ 与 $y_i=f\left(x_i\right)$ 。 $L_1=\left\{x,{(\cdot )}^2,\cdots ,{(\cdot )}^9\right\}$ 与 $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ 。T 表示真， $\mathrm{F}$ 表示假。

Figure 4: Result of linear SR for the Nguyen-1 benchmark,i.e., $f\left(x\right)=x+x^2+x^3$ . Red points represent (test) data set. The red curve represents the true function. The blue and black dashed curves represent the learned functions using $L_1$ and $L_2$ ,respectively.
图 4：Nguyen-1 基准测试的线性 SR 结果，即 $f\left(x\right)=x+x^2+x^3$ 。红点代表（测试）数据集。红色曲线表示真实函数。蓝色和黑色虚线曲线分别表示使用 $L_1$ 和 $L_2$ 学习到的函数。

For mixed polynomial and trigonometric expressions, both library choices do not produce the exact expression. However,a better $R^2$ -coefficient is obtained using $L_1$ . In the case of Nguyen-5 benchmark for example,i.e., $f\left(x\right)=\sin \left(x^2\right)\cos \left(x\right)-1$ ,the resulting function is the Taylor expansion of $f$ :
对于混合多项式和三角表达式，两种库选择均未生成精确表达式。然而，使用 $L_1$ 可获得更优的 $R^2$ 系数。以 Nguyen-5 基准测试为例，即 $f\left(x\right)=\sin \left(x^2\right)\cos \left(x\right)-1$ ，所得函数为 $f$ 的泰勒展开式：

In conclusion, this approach can not learn the ground-truth function when the latter is a multiplication of two functions (i.e., $f\left(x\right)=f_1\left(x\right)\ast f_2\left(x\right)$ ) or when it has a multiplicative or an additive factor to the variable (e.g., $\sin \left(\alpha +x\right),\exp \left(\lambda \ast x\right)$ ,etc.). In the best case,it outputs an approximation of the ground-truth function. Furthermore, this approach fails to predict the correct mathematical expression when the library is extended to include a mixture of polynomial, trigonometric, exponential, and logarithmic functions.
综上所述，当真实函数为两个函数的乘积（即 $f\left(x\right)=f_1\left(x\right)\ast f_2\left(x\right)$ ）或对变量具有乘法或加法因子时（例如 $\sin \left(\alpha +x\right),\exp \left(\lambda \ast x\right)$ 等），该方法无法学习到真实函数。在最佳情况下，它仅能输出真实函数的一个近似。此外，当函数库扩展至包含多项式、三角函数、指数函数和对数函数的混合时，该方法无法预测出正确的数学表达式。

For a given data set $\mathcal{D}=\left\{x_i\in {\mathbb{R}}^d;y_i=f\left(x_1,\cdots ,x_d\right)\right\}$ ,where $d$ is the number of features,the same equations presented in Section 4.1.1 are applicable. However, the dimension of the library matrix U changes to consider the features vector dimension. For example, for the same library shown in Eq. 13 and a two dimensional features
对于给定的数据集 $\mathcal{D}=\left\{x_i\in {\mathbb{R}}^d;y_i=f\left(x_1,\cdots ,x_d\right)\right\}$ ，其中 $d$ 表示特征数量，第 4.1.1 节中提出的方程同样适用。然而，库矩阵 U 的维度需调整为考虑特征向量的维度。例如，对于式 13 所示的相同库及二维特征

vector,i.e., $\mathrm{X}\in {\mathbb{R}}^2,\mathrm{U}\left(\mathrm{X}\right)$ becomes:
向量，即 $\mathrm{X}\in {\mathbb{R}}^2,\mathrm{U}\left(\mathrm{X}\right)$ 变为：

Here, ${\mathbf{X}}^{P_q}$ denotes polynomials in $\mathrm{X}$ of the order $q$ .
此处， ${\mathbf{X}}^{P_q}$ 表示 $\mathrm{X}$ 中阶数为 $q$ 的多项式。

Table 3 presents the results of the experiments performed on two-variables dependent functions,i.e., $f\left(x_1,x_2\right)$ . Similarly to Section 4.1.1, training and test data sets are generated by randomly sampling twenty pairs of points $\left(x_1,x_2\right)$ from a uniform distribution $\mathrm{U}\left(-1,1\right)$ such that $\mathcal{D}={\left\{(x_{1i},x_{2i},f(x_{1i},x_{2i}))\right\}}_{i=1}^n$ . The same choices for the library are considered: $L_1=\left\{x,{(\cdot )}^2,\cdots ,{(\cdot )}^9\right\}$ and $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ . An exact match between the ground-truth and predicted function is obtained using $L_1$ for any polynomial function, whereas only approximate solutions are obtained for trigonometric functions. The results are approximate of the ground-truth function using $L_2$ .
表 3 展示了针对双变量依赖函数（即 $f\left(x_1,x_2\right)$ ）的实验结果。与第 4.1.1 节类似，训练和测试数据集通过从均匀分布 $\mathrm{U}\left(-1,1\right)$ 中随机采样二十对点 $\left(x_1,x_2\right)$ 生成，且满足 $\mathcal{D}={\left\{(x_{1i},x_{2i},f(x_{1i},x_{2i}))\right\}}_{i=1}^n$ 。库的选择保持不变： $L_1=\left\{x,{(\cdot )}^2,\cdots ,{(\cdot )}^9\right\}$ 和 $L_2=L_1\cup \{\sin (\cdot ),\cos (\cdot ),\tan (\cdot ),\exp (\cdot ),\mathrm{sigmoid}(\cdot )\}$ 。对于任何多项式函数，使用 $L_1$ 可得到真实函数与预测函数的精确匹配，而三角函数仅能获得近似解。结果通过 $L_2$ 对真实函数进行了近似。