[a.garcez@city.ac.uk, mxgori@gmail.com, luislamb@acm.org, serafini@fbk.eu, michael.spranger@gmail.com, sn.tran@utas.edu.au]Artur d’Avila Garcez, Marco Gori, Luis C. Lamb, Luciano Serafini, Michael Spranger, Son N. Tran††thanks: Corresponding author. Authors are in alphabetical order.
感谢：通讯作者。作者按字母顺序排列。
[a.garcez@city.ac.uk, mxgori@gmail.com, luislamb@acm.org, serafini@fbk.eu, michael.spranger@gmail.com, sn.tran@utas.edu.au]阿图尔·达维拉·加尔塞兹，马可·戈里，路易斯·C·兰布，卢西亚诺·塞拉菲尼，迈克尔·斯普朗格，孙·N·特兰 † City, Univ. of London, Univ. of Siena, UFRGS, FBK, Sony Japan, Univ. of Tasmania
伦敦城市大学，锡耶纳大学，巴西联邦大学里约格兰德分校，FBK，索尼日本，塔斯马尼亚大学
We thank Richard Evans for his valuable comments and suggestions.
我们感谢 Richard Evans 对他宝贵的评论和建议。
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning
神经符号计算：机器学习和推理原则集成的有效方法论
Abstract 摘要
Current advances in Artificial Intelligence and machine learning in general, and deep learning in particular have reached unprecedented impact not only across research communities, but also over popular media channels. However, concerns about interpretability and accountability of AI have been raised by influential thinkers. In spite of the recent impact of AI, several works have identified the need for principled knowledge representation and reasoning mechanisms integrated with deep learning-based systems to provide sound and explainable models for such systems. Neural-symbolic computing aims at integrating, as foreseen by Valiant, two most fundamental cognitive abilities: the ability to learn from the environment, and the ability to reason from what has been learned. Neural-symbolic computing has been an active topic of research for many years, reconciling the advantages of robust learning in neural networks and reasoning and interpretability of symbolic representation. In this paper, we survey recent accomplishments of neural-symbolic computing as a principled methodology for integrated machine learning and reasoning. We illustrate the effectiveness of the approach by outlining the main characteristics of the methodology: principled integration of neural learning with symbolic knowledge representation and reasoning allowing for the construction of explainable AI systems. The insights provided by neural-symbolic computing shed new light on the increasingly prominent need for interpretable and accountable AI systems.
目前人工智能和机器学习的发展，特别是深度学习的发展，不仅在研究界取得了前所未有的影响，而且在流行媒体渠道上也产生了影响。然而，一些有影响力的思想家提出了关于人工智能可解释性和问责性的担忧。尽管人工智能最近取得了重大进展，但一些研究指出，需要将基于深度学习系统的原则性知识表示和推理机制相结合，为这些系统提供稳健且可解释的模型。神经符号计算旨在整合，正如 Valiant 所预见的那样，两种最基本的认知能力：从环境中学习的能力和从所学内容推理的能力。神经符号计算多年来一直是研究的热门话题，调和了神经网络中强大学习的优势和符号表示中的推理和可解释性。本文概述了神经符号计算作为一种原则性方法论，用于整合机器学习和推理的最新成就。 我们通过概述方法论的主要特征来说明该方法的有效性：将神经学习与符号知识表示和推理原则性地结合，从而实现可解释的人工智能系统的构建。神经符号计算提供的见解为越来越突出的可解释和可问责的人工智能系统的需求带来了新的启示。
1 Introduction 1 介绍
Current advances in Artificial Intelligence (AI) and machine learning in general, and deep learning in particular have reached unprecedented impact not only within the academic and industrial research communities, but also among popular media channels. Deep learning researchers have achieved groundbreaking results and built AI systems that have in effect rendered new paradigms in areas such as computer vision, game
playing, and natural language processing
[27, 45].
Nonetheless, the impact of deep learning has been so remarkable that
leading entrepreneurs such as Elon Musk and Bill Gates, and outstanding scientists such as Stephen Hawking have voiced strong concerns about AI’s accountability, impact on humanity and even on the future of the planet [40].
当前人工智能（AI）和机器学习的最新进展，尤其是深度学习，不仅在学术和工业研究界取得了前所未有的影响，而且在流行媒体渠道中也引起了关注。深度学习研究人员取得了突破性成果，并构建了在计算机视觉、游戏玩法和自然语言处理等领域产生新范式的人工智能系统。然而，深度学习的影响如此显著，以至于像埃隆·马斯克和比尔·盖茨这样的领先企业家，以及像斯蒂芬·霍金这样的杰出科学家都对人工智能的问责制、对人类的影响甚至对地球未来的影响表示了强烈关切。
Against this backdrop, researchers have recognised the need for offering a better understanding of the underlying principles of AI systems, in particular those based on machine learning, aiming at establishing solid foundations for the field. In this respect, Turing Award Winner Leslie Valiant had already pointed out that one of the key challenges for AI in the coming decades is the development of integrated reasoning and learning mechanisms, so as to construct a rich semantics of intelligent cognitive behavior [54]. In Valiant’s words:
“The aim here is to identify a way of looking at and manipulating commonsense
knowledge that is consistent with and can support what we consider to be the
two most fundamental aspects of intelligent cognitive behavior: the ability to learn
from experience, and the ability to reason from what has been learned. We are
therefore seeking
a semantics of knowledge that can computationally support the
basic phenomena of intelligent behavior."
In order to respond to these scientific, technological and societal challenges which demand reliable, accountable and explainable AI systems and tools, the integration of cognitive abilities ought to be carried out in a principled way.
在这种背景下，研究人员已经意识到有必要更好地理解人工智能系统的基本原理，特别是那些基于机器学习的系统，旨在为该领域奠定坚实的基础。在这方面，图灵奖获得者莱斯利·瓦利安特已经指出，未来几十年人工智能面临的关键挑战之一是发展集成推理和学习机制，以建立智能认知行为的丰富语义。瓦利安特说：“这里的目标是确定一种观察和操纵常识知识的方式，这种方式与我们认为是智能认知行为的两个最基本方面相一致并支持：从经验中学习的能力和从所学到的内容进行推理的能力。因此，我们正在寻求一种知识语义，可以在计算上支持智能行为的基本现象。”为了应对这些需要可靠、负责和可解释的人工智能系统和工具的科学、技术和社会挑战，认知能力的整合应该以原则性的方式进行。
Neural-symbolic computing aims at integrating, as put forward by Valiant, two most fundamental cognitive abilities: the ability to learn from experience, and the ability to reason from what has been learned [2, 12, 16].
The integration of learning and reasoning through neural-symbolic computing has been an active branch of AI research for several years [14, 16, 17, 21, 25, 42, 53]. Neural-symbolic computing aims at reconciling the dominating symbolic and connectionist paradigms of AI under a principled foundation.
In neural-symbolic computing, knowledge is represented in symbolic form, whereas learning and reasoning are computed by a neural network. Thus, the underlying characteristics of neural-symbolic computing allow the principled combination of robust learning and efficient inference in neural networks, along with interpretability offered by symbolic knowledge extraction and reasoning with logical systems.
神经符号计算旨在整合，正如 Valiant 所提出的，两种最基本的认知能力：从经验中学习的能力和根据所学知识进行推理的能力。通过神经符号计算整合学习和推理的研究已经成为人工智能研究的一个活跃分支多年。神经符号计算旨在在一个原则性基础下调和人工智能的主导符号和连接主义范式。在神经符号计算中，知识以符号形式表示，而学习和推理则由神经网络计算。因此，神经符号计算的基本特征允许在神经网络中将强大的学习和高效的推理原则性地结合起来，同时结合了符号知识提取和逻辑系统推理所提供的可解释性。
Importantly, as AI systems started to outperform humans in certain tasks [45], several ethical and societal concerns were raised [40]. Therefore, the interpretability and explainability of AI systems become crucial alongside their accountability.
重要的是，随着人工智能系统开始在某些任务中胜过人类[45]，引发了一些伦理和社会关注[40]。因此，人工智能系统的可解释性和可解释性变得至关重要，与其责任相辅相成。
In this paper, we survey the principles of neural-symbolic integration by highlighting key characteristics that underline this research paradigm.
Despite their differences, both the symbolic and connectionist paradigms, share common characteristics offering benefits when integrated in a principled way (see e.g. [8, 16, 46, 53]).
For instance, neural learning and inference under uncertainty may address the brittleness of symbolic systems. On the other hand, symbolism provides additional knowledge for learning which may e.g. ameliorate neural network’s well-known catastrophic forgetting or difficulty with extrapolating.
In addition, the integration of neural models with logic-based symbolic models provides an AI system capable of bridging lower-level information processing (for perception and pattern recognition) and higher-level abstract knowledge (for reasoning and explanation).
在本文中，我们通过突出强调支撑这一研究范式的关键特征，对神经符号整合的原则进行了调查。尽管符号和连接主义范式存在差异，但它们共享共同特征，在以原则方式整合时提供了好处（参见例如[8, 16, 46, 53]）。例如，神经学习和在不确定性下的推理可以解决符号系统的脆弱性。另一方面，符号主义为学习提供了额外的知识，例如可以改善神经网络的众所周知的灾难性遗忘或难以推断的问题。此外，将神经模型与基于逻辑的符号模型整合，提供了一个能够连接低层信息处理（用于感知和模式识别）和高层抽象知识（用于推理和解释）的人工智能系统。
In what follows, we review the important and recent developments of research on neural-symbolic systems.
We start by outlining the main important characteristics of a neural-symbolic system: Representation, Extraction, Reasoning and Learning [2, 17], and their applications. We then discuss and categorise the approaches to representing symbolic knowledge in neural-symbolic systems into three main groups: rule-based, formula-based and embedding-based. After that, we show the capabilities and applications of neural-symbolic systems for learning, reasoning, and explainability. Towards the end of the paper we outline recent trends and identify a few challenges for neural-symbolic computing research.
在接下来的内容中，我们回顾了神经符号系统研究的重要和最新发展。我们首先概述了神经符号系统的主要重要特征：表示、提取、推理和学习[2, 17]，以及它们的应用。然后，我们讨论并将神经符号系统中表示符号知识的方法归类为三大主要群体：基于规则、基于公式和基于嵌入。之后，我们展示了神经符号系统在学习、推理和可解释性方面的能力和应用。在论文的最后，我们概述了神经符号计算研究的最新趋势并确定了一些挑战。
2 Prolegomenon to Neural-Symbolic Computing
2 神经符号计算的序言
Neural-symbolic systems have been applied successfully to several fields, including data science, ontology learning, training and assessment in simulators, and models of cognitive learning and reasoning [5, 14, 16, 34].
However, the recent impact of deep learning in vision and language processing and the growing complexity of
(autonomous) AI systems demand improved explainability and accountability.
In neural-symbolic computing, learning, reasoning and knowledge extraction are combined. Neural-symbolic systems are modular and seek to have the property of compositionality. This is achieved through the streamlined representation of several knowledge representation languages which are computed by connectionist models. The
Knowledge-Based Artificial Neural Network (KBANN) [49] and the Connectionist inductive learning and logic programming (CILP) [17] systems were some of the most influential models that combine logical reasoning and neural learning.
As pointed out in [17] KBANN served as inspiration in the construction of the CILP
system. CILP provides a sound theoretical foundation to inductive learning and reasoning in artificial neural networks through theorems showing how logic programming can be a knowledge representation language for neural networks.
The KBANN system was the first to allow for learning with background knowledge in neural networks and knowledge extraction, with relevant applications in bioinformatics. CILP allowed for the integration of learning, reasoning and knowledge extraction in recurrent networks.
An important result of CILP was to show how neural networks endowed with semi-linear neurons approximate the fixed-point operator of propositional logic programs with negation. This result allowed applications of reasoning and learning using backpropagation and logic programs as background knowledge [17].
神经符号系统已成功应用于多个领域，包括数据科学、本体学习、模拟器培训和评估，以及认知学习和推理模型。然而，深度学习在视觉和语言处理方面的最近影响以及（自主）人工智能系统日益复杂的需求改进了可解释性和问责制。在神经符号计算中，学习、推理和知识提取被结合起来。神经符号系统是模块化的，并且力求具有组合性质。这是通过对由连接主义模型计算的多种知识表示语言的简化表示来实现的。基于知识的人工神经网络（KBANN）和连接主义归纳学习与逻辑编程（CILP）系统是一些结合逻辑推理和神经学习的最具影响力的模型。正如在[17]中指出的那样，KBANN 在构建 CILP 系统时起到了启发作用。 CILP 通过定理展示逻辑编程如何成为神经网络的知识表示语言，为归纳学习和推理提供了坚实的理论基础。KBANN 系统是第一个允许在神经网络中利用背景知识进行学习和知识提取的系统，在生物信息学中具有相关应用。CILP 允许在循环网络中整合学习、推理和知识提取。CILP 的一个重要结果是展示了神经网络配备半线性神经元如何逼近命题逻辑程序的带否定的不动点运算符。这一结果使得可以利用反向传播和逻辑程序作为背景知识进行推理和学习。
Notwithstanding, the need for richer cognitive models soon demanded the representation and learning of other forms of reasoning, such as temporal reasoning, reasoning about uncertainty, epistemic, constructive and argumentative reasoning [16, 54]. Modal and temporal logic have achieved first class status in the formal toolboxes of AI and Computer Science researchers. In AI, modal logics are amongst the most widely used logics in the analysis and modelling of reasoning in distributed multiagent systems.
In the early 2000s, researchers then showed that ensembles of CILP neural networks, when properly set up, can compute the modal fixed-point operator of modal and temporal logic programs. In addition to these results, such ensembles of neural networks were shown to represent the possible world semantics of modal propositional logic, fragments of first order logic and of linear temporal logics.
In order to illustrate the computational power of Connectionist Modal Logics (CML) and Connectionist Temporal Logics of Knowledge (CTLK) [8, 9], researchers were able to learn full solutions to several problems in distributed, multiagent learning and reasoning, including the Muddy Children Puzzle [8] and the Dining Philosophers Problem [26].
尽管如此，对更丰富的认知模型的需求很快就要求表示和学习其他形式的推理，如时间推理、不确定性推理、认识论、建设性和论证性推理。模态逻辑和时间逻辑已经成为人工智能和计算机科学研究人员工具箱中的一流工具。在人工智能领域，模态逻辑是在分布式多智能体系统中推理分析和建模中最广泛使用的逻辑之一。在 21 世纪初，研究人员展示了当适当设置时，CILP 神经网络的集合可以计算模态和时间逻辑程序的模态不动点运算符。除了这些结果外，这些神经网络集合还被证明可以表示模态命题逻辑、一阶逻辑片段和线性时间逻辑的可能世界语义。 为了展示连接主义模态逻辑（CML）和知识连接主义时间逻辑（CTLK）的计算能力，研究人员能够学习到分布式、多 Agent 学习和推理中几个问题的完整解决方案，包括泥泞孩子难题和餐桌上的哲学家问题。
By combining temporal logic with modalities, one can represent knowledge and learning evolution in time. This is a key insight, allowing for temporal evolution of both learning and reasoning in time (see Fig. 1). The Figure represents the integrated learning and reasoning process of CTLK. At each time point (or one state of affairs), e.g. , knowledge which the agents are endowed with and what the agents have learned at the previous time is represented. As time progresses, linear evolution of the agents’ knowledge is represented in time as more knowledge about the world (what has been learned) is represented. Fig. 1 illustrates this dynamic property of CTLK, which allows not only the analysis of the current state of affairs but also of how knowledge and learning evolve over time.
通过将时间逻辑与模态结合，可以在时间中表示知识和学习的演变。这是一个关键的洞察，允许学习和推理在时间中的时间演变（见图 1）。图中表示了 CTLK 的集成学习和推理过程。在每个时间点（或一个事态），例如 ，代理所具有的知识以及代理在上一个时间 学到的知识被表示出来。随着时间的推移，代理的知识的线性演变被表示为更多关于世界的知识（已学到的内容）。图 1 说明了 CTLK 的这种动态特性，它不仅允许分析当前事态，还允许分析知识和学习如何随时间演变。
Modal and temporal reasoning, when integrated with connectionist learning provide neural-symbolic systems with richer knowledge representation languages and better interpretability. As can be seen in Fig. 1, they enable the construction of more modular deep networks. As argued by Valiant, the construction of cognitive models integrating rich logic-based knowledge representation languages, with robust learning algorithms provide an effective alternative to the construction of semantically sound cognitive neural computational models. It is also argued that a language for describing the algorithms of deep neural networks is needed. Non-classical logics such as logic programming in the context of neuro-symbolic systems, and functional languages used in the context of probabilistic programming are two prominent candidates. In the coming sections, we explain how neural-symbolic systems can be constructed from simple definitions which underline the streamlined integration of knowledge representation, learning, and reasoning in a unified model.
模态和时间推理，当与连接主义学习相结合时，为神经符号系统提供了更丰富的知识表示语言和更好的可解释性。如图 1 所示，它们使得更模块化的深度网络的构建成为可能。正如 Valiant 所主张的，将基于丰富逻辑知识表示语言的认知模型与强大的学习算法相结合，提供了一种有效的替代方案，用于构建语义上合理的认知神经计算模型。同时也认为需要一种描述深度神经网络算法的语言。在神经符号系统的背景下，非经典逻辑如逻辑编程，以及在概率编程背景下使用的函数语言是两个突出的候选者。在接下来的章节中，我们将解释如何从简单的定义构建神经符号系统，强调知识表示、学习和推理在统一模型中的流畅整合。
图 1：时间中推理和学习的演变
3 Knowledge Representation in Neural Networks
神经网络中的知识表示
Knowledge representation is the cornerstone of a neural-symbolic
system that provides a mapping mechanism between symbolism and
connectionism, where logical calculus can be carried out exactly or
approximately by a neural network. This way, given a trained neural network,
symbolic knowledge can be extracted for explaining and reasoning purposes. The representation approaches can be categorised into three
main groups: rule-based, formula-based and embedding, which are
discussed as follows.
知识表示是神经符号系统的基石，提供了一种在符号主义和连接主义之间进行映射的机制，逻辑演算可以通过神经网络精确地或近似地进行。这样，给定一个经过训练的神经网络，可以提取符号知识用于解释和推理。表示方法可以分为三大类：基于规则、基于公式和嵌入式，下面将对其进行讨论。
3.1 Propositional Logic 3.1 命题逻辑
3.1.1 Rule-based Representation
3.1.1 基于规则的表示
(a) KBANN（ 表示阈值）。
图 2：使用 KBANN 和 CILP 对 进行知识表示。
Early work on representation of symbolic knowledge in connectionist
networks focused on tailoring the models’ parameters to establish an
equivalence between input-output mapping function of artificial neural
networks (ANN) and logical inference rules. It has been shown that by
constraining the weights of a neural network, inference
with feedforward propagation can exactly imitate the behaviour of
modus ponens [49, 7]. KBANN [49]
employs stack of perceptrons to represent the inference rule of
logical implications. For example, given a set of rules:
早期关于在连接主义网络中表示符号知识的研究侧重于调整模型的参数，以建立人工神经网络（ANN）的输入-输出映射函数与逻辑推理规则之间的等价性。已经证明，通过约束神经网络的权重，使用前馈传播进行推理可以精确模拟假言推理的行为。KBANN [49] 使用感知器堆栈来表示逻辑蕴涵的推理规则。例如，给定一组规则：
| (1) |
an ANN can be constructed as in Figure 2(a). CILP then generalises the
idea by using recurrent networks and bounded continuous units
[7]. This representation method allows the use of various data types and more complex sets of rules. With CILP, knowledge given in
Eq. (1) can be encoded in a neural network as shown in Figure
2(b). In order to adapt this system to first-order logic, CILP++
[15] makes use of techniques from Inductive Logic Programming (ILP). In CILP++, examples and background knowledge are converted into propositional clauses by a bottom-clause propositionalisation technique, which are then encoded into an ANN with recurrent connections as done by CILP.
ANN 可以构建如图 2(a)所示。然后，CILP 通过使用循环网络和有界连续单元推广了这个想法。这种表示方法允许使用各种数据类型和更复杂的规则集。通过 CILP，Eq.（1）中给出的知识可以被编码到神经网络中，如图 2(b)所示。为了将这个系统适应一阶逻辑，CILP++利用归纳逻辑编程（ILP）的技术。在 CILP++中，示例和背景知识通过底层子句命题化技术转换为命题子句，然后像 CILP 一样被编码为具有循环连接的 ANN。
3.1.2 Formula-based Representation
3.1.2 基于公式的表示
（a）用于惩罚逻辑的高阶网络。
(b) 具有置信规则的 RBM。
图 3：使用惩罚逻辑和置信规则表示 的知识
One issue with KBANN-style rule-based representations is that the discriminative structure of ANNs will only allow a subset of the variables (the
consequent of the if-then formula) to be inferred, unless recurrent networks are deployed, with the other variables (the antecedents) being seen as inputs only. This would not represent the
behaviour of logical formulas and does not support general
reasoning where any variable can be inferred. In order to solve this
issue, generative neural networks can be employed as they can treat
all variables as non-discriminative. In this formula-based approach, typically associated with restricted Boltzmann machines (RBMs) as a building block, the focus is on mapping logical formulas to symmetric connectionist networks, each
characterised by an energy function. Early work such as penalty logic
[35] proposes a mechanism to represent weighted formulas
in energy-based connectionist (Hopfield) networks where maximising satisfiability
is equivalent to minimising energy function. Suppose that each
formula in the knowledge base (1) is assigned a weight .
Penalty logic constructs a higher-order Hopfield network as shown
in Figure 3(a). However, inference with such type of network is difficult, while converting the higher-order energy
function to a quadratic form is possible but computationally
expensive. Recent work on confidence rules [51] proposes an
efficient method to represent propositional formulas in restricted
Boltzmann machines and deep belief networks where inference and learning
become easier. Figure 3(b) shows an RBM for the knowledge
base (1). Nevertheless, learning and reasoning with
restricted Boltzmann machines are still complex, making it more difficult to apply formula-based representations than rule-based
representations in practice. The main issue has to do with the
partition functions of symmetric connectionist networks which cannot
be computed analytically. This intractability problem, fortunately,
can be ameliorated using sum-product approach as has been shown
in [38]. However, it is not yet clear how to apply this idea to RBMs.
KBANN 风格的基于规则的表示存在一个问题，即 ANNs 的判别结构只允许推断变量的子集（即 if-then 公式的结论部分），除非部署循环网络，否则其他变量（前提）只被视为输入。这不代表逻辑公式的行为，也不支持任何变量都可以被推断的一般推理。为了解决这个问题，可以使用生成式神经网络，因为它们可以将所有变量视为非判别性的。在这种基于公式的方法中，通常与受限玻尔兹曼机（RBMs）作为构建模块相关联，重点是将逻辑公式映射到对称连接主义网络，每个网络都具有能量函数。早期的工作，如惩罚逻辑[35]，提出了一种在基于能量的连接主义（Hopfield）网络中表示加权公式的机制，其中最大化可满足性等同于最小化能量函数。假设知识库中的每个公式（1）被分配一个权重 。惩罚逻辑构建了一个高阶 Hopfield 网络，如图 3(a)所示。 然而，利用这种类型的网络进行推理是困难的，尽管将高阶能量函数转换为二次形式是可能的，但计算成本很高。最近关于置信规则的研究提出了一种有效的方法，可以在受限玻尔兹曼机和深度信念网络中表示命题公式，从而使推理和学习变得更容易。图 3(b)显示了一个用于知识库(1)的 RBM。然而，使用受限玻尔兹曼机进行学习和推理仍然很复杂，这使得在实践中更难应用基于公式的表示而不是基于规则的表示。主要问题在于对称连接网络的分区函数无法通过解析计算。幸运的是，这种难以处理的问题可以通过和-积方法得到缓解，正如[38]中所示。然而，如何将这个想法应用到 RBM 中尚不清楚。
3.2 First-order Logic 3.2 一阶逻辑
3.2.1 Propositionalisation
3.2.1 命题化
Representation of knowledge in first-order logic in neural networks has been an ongoing challenge, but it can benefit from studies of propositional logic representation 3.1 using propositionalisation techniques [30]. Such techniques allow a first-order knowledge base to be converted into a propositional knowledge base so as to preserve entailment. In neural-symbolic computing, bottom clause prositionalisation (BCP) is a popular approach because bottom clause literals can be encoded directly into neural networks as data features while at the same time presenting semantic meaning.
在神经网络中以一阶逻辑表示知识一直是一个持续的挑战，但可以从命题逻辑表示的研究中受益，使用命题化技术[30]。这些技术允许将一阶知识库转换为命题知识库，以保留蕴涵。在神经符号计算中，底部子句命题化（BCP）是一种流行的方法，因为底部子句文字可以直接编码为神经网络的数据特征，同时呈现语义含义。
Early work from [11] employs prositionalisation and feedforward neural networks to learn a clause evaluation function which helps improve the efficiency in exploring large hypothesis spaces. In this approach, the neural network does not work as a standalone ILP system, instead it is used to approximate clause evaluation scores to decide the direction of the hypothesis search. In [36], prositionalisation is used for learning first-
order logic in Bayesian networks. Inspired by this work, in [15], the CILP++ system is proposed by integrating bottom clauses and rule-based approach CILP [17], referred to in Section 3.1.1.
早期的工作[11]采用命题化和前馈神经网络来学习一个子句评估函数，从而帮助提高探索大假设空间的效率。在这种方法中，神经网络并不是作为独立的 ILP 系统，而是用来近似子句评估分数，以决定假设搜索的方向。在[36]中，命题化被用于在贝叶斯网络中学习一阶逻辑。受到这项工作的启发，在[15]中，通过整合底层子句和基于规则的方法 CILP [17]，在第 3.1.1 节中提到，提出了 CILP++系统。
The main advantage of propositionalisation is that it is efficient and it fits neural networks well. Also, it does not require first-order formulas to be provided as bottom clauses. However, propositionalisation has serious disadvantages. First, with function symbols, there are infinitely many ground terms. Second, propositionalization seems to generate lots of irrelevant clauses.
命题化的主要优势在于它高效且与神经网络很匹配。此外，它不需要将一阶公式提供为底部子句。然而，命题化也有严重的缺点。首先，使用函数符号会产生无限多的基础项。其次，命题化似乎会生成大量无关的子句。
3.2.2 Tensorisation 3.2.2 张量化
图 4：逻辑张量网络，其中 与 和 ； 是第一阶语言符号的基础（向量表示）；本例中的张量阶为 [ 42]。
Tensorisation is a class of approaches that embeds first-order logic symbols such as constants, facts and rules into real-valued tensors. Normally, constants are represented as one-hot vectors (first order tensor). Predicates and functions are matrices (second-order tensor) or higher-order tensors.
张量化是一类方法，将一阶逻辑符号（如常量、事实和规则）嵌入到实值张量中。通常，常量被表示为单热向量（一阶张量）。谓词和函数是矩阵（二阶张量）或更高阶张量。
In early work, embedding techniques were
proposed to transform symbolic representations into vector spaces where
reasoning can be done through matrix computation
[4, 47, 48, 42, 41, 6, 14, 57, 13, 39]. Training
embedding systems can be carried out as distance learning using
backpropagation. Most research in this direction focuses on
representing relational predicates in a neural network. This is known
as "relational embedding"
[4, 41, 47, 48]. For representation
of more complex logical structures, i.e. first order-logic formulas, a
system named Logic Tensor Network (LTN) [42] is
proposed by extending Neural Tensor Networks (NTN)
[47], a state-of-the-art relational embedding
method. Figure 4 shows an example of LTN for . Related ideas are discussed formally in the context of
constraint-based learning and reasoning [19]. Recent research in first-order logic programs has successfully exploited the advantages of distributed representations of logic symbols for efficient reasoning [6], inductive programming [14, 57, 13], and differentiable theorem proving [39].
在早期的工作中，提出了嵌入技术，将符号表示转换为向量空间，通过矩阵计算进行推理[4, 47, 48, 42, 41, 6, 14, 57, 13, 39]。训练嵌入系统可以通过反向传播进行距离学习。这个方向上的大部分研究集中在在神经网络中表示关系谓词。这被称为“关系嵌入”[4, 41, 47, 48]。为了表示更复杂的逻辑结构，即一阶逻辑公式，提出了一种名为逻辑张量网络（LTN）[42]的系统，通过扩展神经张量网络（NTN）[47]，这是一种最先进的关系嵌入方法。图 4 展示了 LTN 的一个示例 。相关思想在基于约束的学习和推理的背景下进行了正式讨论[19]。最近在一阶逻辑程序方面的研究成功地利用了逻辑符号的分布式表示的优势，用于高效推理[6]、归纳编程[14, 57, 13]和可微定理证明[39]。
3.3 Temporal Logic 3.3 时间逻辑
One of the earliest works on temporal logic and neural networks is CTLK, where ensembles of recurrent neural networks are set up to represent the possible world semantics of linear temporal logics [8]. With single hidden layers and semi-linear neurons, the networks can compute a fixed-point semantics of temporal logic rules. Another work on representation of temporal knowledge is proposed in Sequential Connectionist Temporal Logic (SCTL) [5] where CILP is extended to work with the nonlinear auto-regressive exogenous NARX network model. Neural-Symbolic Cognitive Agents (NSCA) represent temporal knowledge in recurrent temporal RBMs [34]. Here, the temporal logic rules are modelled in the form of recursive conjunctions represented by recurrent structures of RBMs. Temporal relational knowledge embedding has been studied recently in Tensor Product Recurrent Neural Network (TPRN) with applications to
question-answering [32].
在时间逻辑和神经网络领域的早期作品之一是 CTLK，其中建立了一组递归神经网络来表示线性时间逻辑的可能世界语义[8]。通过单隐藏层和半线性神经元，这些网络可以计算时间逻辑规则的不动点语义。另一项关于时间知识表示的工作是在顺序连接主义时间逻辑（SCTL）中提出的[5]，其中将 CILP 扩展到与非线性自回归外生 NARX 网络模型一起工作。神经符号认知代理（NSCA）在递归时间 RBM 中表示时间知识[34]。在这里，时间逻辑规则以由 RBM 的递归结构表示的递归连接的形式进行建模。最近在张量积递归神经网络（TPRN）中研究了时间关系知识嵌入，应用于问答[32]。
4 Neural-Symbolic Learning
4 神经符号学习
4.1 Inductive Logic Programming
4.1 归纳逻辑编程
Inductive logic programming (ILP) can take advantage of the learning capability of neural-symbolic computing to automatically construct a logic program from examples. Normally, approaches in ILP are categorised into bottom-up and top-down which inspire the development of neural-symbolic approaches accordingly for learning logical rules.
归纳逻辑编程（ILP）可以利用神经符号计算的学习能力，从示例中自动构建逻辑程序。通常，ILP 中的方法被归类为自下而上和自上而下，相应地激发了神经符号方法的发展，用于学习逻辑规则。
Bottom-up approaches construct logic programs by extracting specific clauses from examples. After that, generalisation procedures are usually applied to search for more general clauses. This is well suited to the idea of propositionalisation discussed earlier in Section 3.2.1. For example, CILP++ [15] employed a bottom clause propositionalisation technique to construct CILP++. In [52], a system called CRILP is proposed by integrating bottom clauses generated from [15] with RBMs. However, both CILP++ and CRILP learn and fine-tune formulas at a propositional level where propositionalisation would generate a large number of long clauses resulting in very large networks. This leaves an open research question of generalising bottom clauses within neural networks that scale well and can extrapolate.
自下而上的方法通过从示例中提取特定子句来构建逻辑程序。之后，通常会应用泛化程序来搜索更一般的子句。这与前面在第 3.2.1 节中讨论的命题化的概念非常契合。例如，CILP++ [15]采用了一种底部子句命题化技术来构建 CILP++。在[52]中，提出了一个名为 CRILP 的系统，该系统通过将从[15]生成的底部子句与 RBMs 集成来构建。然而，无论是 CILP++还是 CRILP 都是在命题级别学习和微调公式，其中命题化会生成大量长子句，导致网络非常庞大。这引出了一个开放的研究问题，即如何在神经网络中泛化底部子句，使其具有良好的可扩展性和外推能力。
Top-down approaches, on the other hand, construct logic programs from the most general clauses and extend them to be more specific. In neural-symbolic terms, the most popular idea is to take advantage of neural networks’ learning and inference capabilities to fine-tune and test the quality of rules. This can be done by replacing logical operations by differentiable operations. For example, in Neural Logic Programming (NLP) [57], learning of rules are based on the differentiable inference of TensorLog [6]. Here, matrix computations are used to soften logic operators where the confidence of conjunctions and confidence of disjunctions are computed as product and sum, respectively. NLP generate rules from facts, starting with the most general ones. In
Differentiable Inductive Logic Programming (ILP) [14], rules are generated from templates, which are assigned to parameters (weights) to make the loss function between actual conclusions and predicted conclusions from forward chaining differentiable.
In [39], Neural Theorem Prover (NTP) is proposed by extending the backward chaining method to be differentiable. It shows that latent predicates from rule templates can be learned through optimisation of their distributed representations. Different from [57, 14, 39] where clauses are generated and then softened by neural networks, in Neural Logic Machines (NLM) [13] the relation of predicates is learned by a neural network where input tensors represent facts (predicates of different arities) from a knowledge base and output tensors represent new facts.
自上而下的方法则是从最一般的子句构建逻辑程序，并将其扩展为更具体的形式。在神经符号术语中，最流行的想法是利用神经网络的学习和推理能力来微调和测试规则的质量。这可以通过用可微操作替换逻辑操作来实现。例如，在神经逻辑编程（NLP）[57]中，规则的学习基于 TensorLog [6]的可微推理。在这里，矩阵计算用于软化逻辑运算符，其中合取的置信度和析取的置信度分别计算为乘积和总和。NLP 从事实中生成规则，从最一般的规则开始。在可微归纳逻辑编程（ ILP）[14]中，规则是从模板生成的，这些模板被分配给参数（权重），以使实际结论与正向链接的预测结论之间的损失函数可微。在[39]中，通过将向后链接方法扩展为可微，提出了神经定理证明器（NTP）。 它表明，通过优化其分布式表示，可以学习规则模板中的潜在谓词。与[57, 14, 39]不同，在神经逻辑机（NLM）[13]中，谓词的关系是通过神经网络学习的，其中输入张量表示来自知识库的事实（不同阶数的谓词），输出张量表示新事实。
4.2 Horizontal Hybrid Learning
4.2 水平混合学习
Effective techniques such as deep learning usually require large
amounts of data to exhibit statistical regularities. However, in many
cases where collecting data is difficult a small dataset would make
complex models more prone to overfitting. When prior knowledge is
provided, e.g. from domain experts, a neural-symbolic system can offer
the advantage of generality by combining logical rules/formulas with
data during learning, while at the same time using the data to
fine-tune the knowledge. It has been shown that encoding knowledge
into a neural network can result in
performance improvements [7, 12, 49, 52]. Also, it is evident that using symbolic
knowledge can help improve the efficiency of neural network learning [7, 15]. Such effectiveness and
efficiency are obtained by encoding logical knowledge as controlled
parameters during the training of a model. This technique, in general terms, has been known as learning with logical
constraints [19]. Besides, in the case of lacking
prior knowledge one can apply the idea of neural-symbolic integration
for knowledge transfer learning [51]. The idea is
to extract symbolic knowledge from a related domain and transfer it to
improve the learning in another domain, starting from a network that does not necessarily have to be instilled with background knowledge. Self-transfer with
symbolic-knowledge distillation [23] is also useful
as it can enhance several types of deep networks such as convolutional
neural networks and recurrent neural networks. Here, symbolic
knowledge is extracted from a trained network called “teacher” which
then would be used to encoded as regularizers to train a “student”
network in the same domain.
有效的技术，如深度学习，通常需要大量数据来展示统计规律。然而，在许多情况下，收集数据困难，小数据集会使复杂模型更容易过拟合。当提供先验知识时，例如来自领域专家，神经符号系统可以通过在学习过程中将逻辑规则/公式与数据结合起来，从而提供广泛性的优势，同时利用数据来微调知识。已经证明，将知识编码到神经网络中可以提高性能。此外，使用符号知识可以帮助提高神经网络学习的效率。通过在模型训练过程中将逻辑知识编码为受控参数，可以获得这种有效性和效率。这种技术，一般来说，被称为具有逻辑约束的学习。此外，在缺乏先验知识的情况下，可以应用神经符号整合的思想进行知识迁移学习。 这个想法是从一个相关领域中提取符号知识，并将其转移以改善另一个领域中的学习，从一个不一定需要灌输背景知识的网络开始。具有符号知识蒸馏的自我转移[23]也很有用，因为它可以增强多种类型的深度网络，如卷积神经网络和循环神经网络。在这里，符号知识是从一个称为“教师”的训练网络中提取出来，然后被用作正则化器来训练同一领域中的“学生”网络。
4.3 Vertical Hybrid Learning
4.3 垂直混合学习
Studies in neuroscience
show that some areas in the brain are used for processing input
signals e.g. visual cortices for images [20, 37],
while other areas are responsible for logical thinking and reasoning
[43]. Deep neural networks can learn high level
abstractions from complex input data such as images, audio, and text, which should be useful at making decisions. However, despite that
optimisation process during learning being mathematically justified, it is
difficult for humans to comprehend how a decision has been made during inference time. Therefore, placing a logic
network on top of a deep neural network to learn the relations of those
abstractions, can help the
system to be able to explain itself. In [12], a Fast-RCNN [18] is used for bounding-box detection of parts of objects and on top of that, a Logic Tensor Network is used to reason about relations between parts of objects and types of such objects. In such work, the perception part (Fast-RCNN) is fixed and learning is carried out in the reasoning part (LTN). In a related approach, called DeepProbLog, end-to-end learning and reasoning have been studied [28] where outputs of neural networks are used as "neural predicates" for ProbLog [10].
神经科学研究表明，大脑的某些区域用于处理输入信号，例如视觉皮层用于图像[20, 37]，而其他区域负责逻辑思维和推理[43]。深度神经网络可以从复杂的输入数据（如图像、音频和文本）中学习高层抽象，这对于做出决策应该是有用的。然而，尽管在学习过程中的优化过程在数学上是合理的，但人类很难理解在推理时是如何做出决策的。因此，在深度神经网络顶部放置逻辑网络来学习这些抽象的关系，可以帮助系统能够解释自己。在[12]中，使用 Fast-RCNN[18]来进行对象部分的边界框检测，然后在此基础上使用逻辑张量网络来推理对象部分之间的关系和这些对象的类型。在这样的工作中，感知部分（Fast-RCNN）是固定的，学习是在推理部分（LTN）进行的。 在一种相关的方法中，称为 DeepProbLog，已经研究了端到端的学习和推理[28]，其中神经网络的输出被用作 ProbLog 的“神经谓词”[10]。
5 Neural-symbolic Reasoning
5 神经符号推理
Reasoning is an important feature of a neural-symbolic system and has recently
attracted much attention from the research community [14]. Various attempts have
been made to perform reasoning within neural networks, both
model-based and theorem proving approaches. In neural-symbolic
integration the main focus is the integration of reasoning and
learning, so that a model-based approach is preferred. Most theorem
proving systems based on neural networks, including first-order logic
reasoning systems such as SHRUTI [56], have been unable to perform
learning as effectively as end-to-end differentiable learning
systems. On the other hand, model-based approaches have been shown
implementable in neural networks in the case of nonmonotonic,
intuitionistic and propositional modal logic, as well as abductive reasoning and other forms of human reasoning [2, 5]. As a result, the focus of
neural-symbolic computation has changed from performing
symbolic reasoning in neural networks, such as for example
implementing the logical unification algorithm in a neural network, to
the combination of learning and reasoning, in some cases with
a much more loosely-defined approach rather than full integration,
whereby a hybrid system will contain different components which may be
neural or symbolic and which communicate with each other.
推理是神经符号系统的一个重要特征，最近引起了研究界的广泛关注。已经尝试了各种方法在神经网络内进行推理，包括基于模型和定理证明方法。在神经符号整合中，主要关注推理和学习的整合，因此更倾向于采用基于模型的方法。大多数基于神经网络的定理证明系统，包括一阶逻辑推理系统如 SHRUTI，未能像端到端可微分学习系统那样有效地进行学习。另一方面，基于模型的方法已被证明可以在神经网络中实现，例如非单调、直觉主义和命题模态逻辑，以及归纳推理和其他形式的人类推理。 因此，神经符号计算的重点已经从在神经网络中执行符号推理，例如在神经网络中实现逻辑统一算法，转变为学习和推理的结合，在某些情况下采用更宽松定义的方法，而不是完全集成，混合系统将包含不同的组件，这些组件可能是神经的或符号的，并彼此通信。
5.1 Forward and Backward chaining
5.1 前向和后向链接
Forward chaining and backward chaining are two popular inference techniques for logic programs and other logical systems. In the case of neural-symbolic systems forward and backward chainings are both in general implemented by feedforward inference.
前向链接和后向链接是逻辑程序和其他逻辑系统中两种流行的推理技术。在神经符号系统中，前向链接和后向链接通常都是通过前馈推理实现的。
Forward chaining generates new facts from the head literals of the rules using existing facts in the knowledge base. For example, in [34], a
“Neural-symbolic Cognitive Agent” shows that it is possible to
perform online learning and reasoning in real-world scenarios, where
temporal knowledge can be extracted to reason about driving skills [34]. This can be seen as forward chaining over time. In ILP [14], a differentiable function is defined for each clause to carry out a single step of forward chaining. Similar to this, NLM [13] employs neural networks as a differentiable chain for forward inference. Different from ILP, NLM represent the outputs and inputs of neural networks as grounding tensors of predicates for existing facts and new facts respectively.
前向链接从规则的头文字面生成新事实，使用知识库中的现有事实。例如，在[34]中，“神经符号认知代理”表明可以在现实场景中执行在线学习和推理，其中可以提取时间知识来推理驾驶技能[34]。这可以被视为随时间的前向链接。在 ILP [14]中，为每个子句定义了一个可微函数，以执行前向链接的单个步骤。类似于此，NLM [13]将神经网络作为前向推理的可微链。与 ILP 不同，NLM 将神经网络的输出和输入表示为现有事实和新事实的基础张量。
Backward chaining, on the other hand, searches backward from a goal in the knowledge base to determine whether a query is derivable or not. This form a tree search starts from the query and expands further to the literals in the body of the rules whose heads match the query. TensorLog [6] implements backward chaining using neural networks as symbols. The idea is based on stochastic logic programs [31], and soft logic is applied to transform the hypothesis search into a chain of matrix operations. In NTP, a neural system is constructed recursively for backward chaining and unification where AND and OR operators are represented as networks. In general, backward (goal-directed) reasoning is considerably harder to achieve in neural networks than forward reasoning. This is another current line of research within neuro-symbolic computation and AI.
反向链接，另一方面，从知识库中的目标向后搜索，以确定查询是否可推导。这种树搜索从查询开始，进一步扩展到规则主体中与查询匹配的文字。TensorLog [6] 使用神经网络作为符号实现反向链接。这个想法基于随机逻辑程序[31]，并且应用软逻辑将假设搜索转换为一系列矩阵操作。在 NTP 中，为反向链接和统一递归构建了一个神经系统，其中 AND 和 OR 运算符表示为网络。一般来说，在神经网络中实现反向（目标导向）推理比实现正向推理要困难得多。这是神经符号计算和人工智能中另一个当前的研究方向。
5.2 Approximate Satisfiability
5.2 近似可满足
Inference in the case of logic programs with arbitrary formulas is more complex. In general, one may want to search over the hypothesis space to find a solution that satisfies (mostly) the formulas and facts in the knowledge base. Exact inference, that is, reasoning maximising satisfiability, is NP-hard. For this reason, some neural-symbolic systems offer a mechanism of
approximate satisfiability. Tensor logic networks are trained
to approximate the best satisfiability [42] making
inference efficient with feedforward propagation. This has made LTNs applicable successfully to the Pascal data set and image understanding
[12]. Penalty logic shows an equivalence between minimising violation and minimising energy functions of symmetric connectionist networks [35]. Confidence rules, another
approximation approach, shows the relation between sampling in
restricted Boltzmann machines and search for truth-assignments which
maximise satisfiability. The use of confidence rules also allows one to measure how confident a neural network is in its own answers. Based on that, neural-symbolic system
“confidence rule inductive logic programming (CRILP)” was constructed and applied to inductive logic
programming [52].
在具有任意公式的逻辑程序的推理更加复杂。一般来说，人们可能希望在假设空间中搜索，以找到一个满足（大部分）知识库中的公式和事实的解决方案。精确推理，即最大可满足性推理，是 NP 难的。因此，一些神经符号系统提供了一种近似可满足性的机制。张量逻辑网络被训练为近似最佳可满足性[42]，通过前向传播使推理高效。这使得 LTNs 成功应用于 Pascal 数据集和图像理解[12]。惩罚逻辑展示了最小化违反和最小化对称连接网络的能量函数之间的等价性[35]。置信规则，另一种近似方法，展示了在受限玻尔兹曼机中进行采样与搜索最大化可满足性的真值赋值之间的关系。置信规则的使用还允许人们衡量神经网络对自己答案的信心。 基于这一点，构建了神经符号系统“置信规则归纳逻辑编程（CRILP）”，并应用于归纳逻辑编程[52]。
5.3 Relationship reasoning
5.3 关系推理
Relational embedding systems have been used for reasoning about
relationships between entities. Technically, this has been done by
searching for the answer to a query that gives the highest grounding
score [4, 3, 47, 48]. Deep neural networks are also employed for visual reasoning where they learn and infer relationships and features of multiple objects in images [41, 58, 29].
关系嵌入系统已被用于推理实体之间的关系。技术上，这是通过搜索给出最高基础分数的查询答案来实现的[4, 3, 47, 48]。深度神经网络也被用于视觉推理，它们学习和推断图像中多个对象的关系和特征[41, 58, 29]。
6 Neural-symbolic Explainability
6 神经符号解释能力
The (re)emergence of deep networks has again raised the question of explainability. The complex structure of a deep neural network turns
them into a powerful learning system if one can correctly engineer its components such as type
of hidden units, regularisation and optimization methods. However,
limitations of some AI applications have heightened the need for
explainability and interpretability of deep neural
networks. More importantly, besides improving deep neural networks for
better applications one should also look for the benefits that deep networks can
offer in terms of knowledge acquisition.
深度网络的（再次）出现再次引发了可解释性的问题。深度神经网络的复杂结构使其成为强大的学习系统，如果能正确地设计其组件，如隐藏单元的类型、正则化和优化方法。然而，一些人工智能应用的局限性增加了对深度神经网络的可解释性和可解释性的需求。更重要的是，除了改进深度神经网络以获得更好的应用之外，人们还应该寻找深度网络在知识获取方面能够提供的好处。
6.1 Knowledge Extraction 6.1 知识提取
Explainability is a promising capability of neural-symbolic systems
where the behaviour of a connectionist network can be represented in a set
of human-readable expressions. In early work, the demand for solving
“black-box” issues of neural networks has motivated a number of
rules extraction methods. Most of them are discussed in the surveys [1, 24, 55]. These attempts were to search for logic rules
from a trained network based on four criteria: (a) accuracy, (b)
fidelity, (c) consistency and (d) comprehensibility
[1]. In [17], a sound extraction approach
based on partially ordered sets is proposed to narrow the search of
logic rules.
However, such combinatorial approaches do not scale well
to deal with the dimensionality of current networks. As a result, gradually less attention has been paid to knowledge
extraction until recently when the combination of global and local approaches started to be investigated. The idea here is either to create modular networks with rule extraction applying to specific modules or to consider rule extraction from specific layers only.
可解释性是神经符号系统的一个有前途的能力，其中连接主义网络的行为可以用一组人类可读的表达来表示。在早期的工作中，解决神经网络的“黑盒”问题的需求激发了许多规则提取方法。其中大部分在调查中进行了讨论[1, 24, 55]。这些尝试是为了基于四个标准（a）准确性，（b）忠实度，（c）一致性和（d）可理解性，从经过训练的网络中搜索逻辑规则[1]。在[17]中，提出了一种基于部分有序集的有效提取方法，以缩小逻辑规则的搜索范围。然而，这种组合方法在处理当前网络的维度方面并不很有效。因此，直到最近，当全局和局部方法的结合开始被研究时，对知识提取的关注逐渐减少。这里的想法是要么创建具有规则提取应用于特定模块的模块化网络，要么仅考虑从特定层提取规则。
In [50, 51],
it has been shown that while extracting conjunctive clauses
from the first layer of a deep belief network is fast and effective,
extraction in higher layers results in a loss of accuracy. A trained
deep network can be employed instead for
extraction of soft-logic rules which is less formal but more flexible [23].
Extraction of temporal rules have been
studied in [34] and generated semantic relations of domain
variables over time. Besides formal logical knowledge, hierarchical
Boolean expressions can be learned from images for object detection and
recognition [44].
在[50, 51]中已经表明，从深度信念网络的第一层提取连接子句是快速且有效的，但在更高层提取会导致准确性下降。可以使用经过训练的深度网络来提取软逻辑规则，这种规则不太正式但更灵活[23]。在[34]中研究了时间规则的提取，并生成了随时间变化的领域变量的语义关系。除了形式逻辑知识外，还可以从图像中学习用于对象检测和识别的分层布尔表达式[44]。
6.2 Natural Language Generation
6.2 自然语言生成
For explainability purposes, another approach
couples a deep network with sequence models to extract natural
language knowledge [22]. In [4],
instead of investigating the parameters of a trained model,
relational knowledge extraction is proposed where predicates
are obtained by performing inference of a trained embedding network on
text data.
为了解释的目的，另一种方法是将深度网络与序列模型相结合，以提取自然语言知识[22]。在[4]中，与研究训练模型的参数不同，提出了关系知识提取的方法，其中通过在文本数据上对训练嵌入网络进行推理来获得谓词。
6.3 Program Synthesis 6.3 程序合成
In the field of Program Induction, neuro-symbolic program synthesis (NSPS) has been proposed to construct computer programs on an incremental fashion using a large amount of input-output samples [33]. A neural network is employed to represent partial trees in a domain-specific language are tree nodes, symbols and rules are vector representations. Explainability can be achieved through the tree-based structure of the network. Again, this shows that the integration of neural networks and symbolic representation is indeed a solution for both scalability and explainability.
在程序归纳领域，神经符号程序合成（NSPS）被提出用于使用大量输入-输出样本逐步构建计算机程序[33]。神经网络被用来表示特定领域语言中的部分树，树节点、符号和规则是向量表示。通过网络的基于树的结构可以实现可解释性。再次表明，神经网络和符号表示的集成确实是可扩展性和可解释性的解决方案。
7 Conclusions 7 结论
In this paper, we highlighted the key ideas and principles of
neural-symbolic computing. In order to do so, we illustrated the main
methodological approaches which allow for the integration of effective
neural learning with sound symbolic-based, knowledge representation
and reasoning methods. One of the principles we highlighted in the
paper is the sound mapping between symbolic rules and neural networks
provided by neural-symbolic computing methods. This mapping allows
several knowledge representation formalisms to be used as background
knowledge for potentially large-scale learning and efficient
reasoning. This interplay between efficient neural learning and
symbolic reasoning opens relevant possibilities towards richer
intelligent systems. The comprehensibility and compositionality of
neural-symbolic systems, offered by building networks with a logical
structure, allows for integrated learning and reasoning under
different logical systems. This opens several interesting research
lines, in which learning is endowed with the sound semantics of
diverse logics. This, in turn, contributes towards the development of
explainable and accountable AI and machine learning-based systems and
tools.
在本文中，我们强调了神经符号计算的关键思想和原则。为了做到这一点，我们阐述了主要的方法论方法，这些方法允许将有效的神经学习与基于符号的知识表示和推理方法相结合。我们在论文中强调的原则之一是神经符号计算方法提供的符号规则与神经网络之间的良好映射。这种映射允许多种知识表示形式主义被用作潜在的大规模学习和高效推理的背景知识。有效神经学习和符号推理之间的相互作用为更丰富的智能系统开辟了相关可能性。神经符号系统的可理解性和组合性，通过构建具有逻辑结构的网络，允许在不同逻辑系统下进行集成学习和推理。这为多个有趣的研究方向打开了大门，其中学习赋予了不同逻辑的良好语义。 这反过来有助于可解释和可追溯的人工智能和基于机器学习的系统和工具的发展。
See nesy.pdf 请查看 nesy.pdf。
References
-
[1]
R. Andrews, J. Diederich, and A. Tickle.
Survey and critique of techniques for extracting rules from trained
artificial neural networks.
Know.-Based Syst., 8(6):373–389, December 1995.
R. Andrews, J. Diederich, 和 A. Tickle. 从训练好的人工神经网络中提取规则的技术调查与评论。知识基础系统，8(6):373–389，1995 年 12 月。 -
[2]
S. Bader and P. Hitzler.
Dimensions of neural-symbolic integration: a structured survey.
In S. Artemov, H. Barringer, A. d’Avila Garcez, L. Lamb, and
J. Woods, editors, We Will Show Them! Essays in Honour of Dov Gabbay,
2005.
S. Bader 和 P. Hitzler。神经符号整合的维度：结构化调查。在 S. Artemov，H. Barringer，A. d'Avila Garcez，L. Lamb 和 J. Woods 编辑的《我们会向他们展示！献给 Dov Gabbay 的论文》，2005 年。 -
[3]
A. Bordes, X. Glorot, J. Weston, and Y. Bengio.
Joint learning of words and meaning representations for open-text
semantic parsing.
In AISTATS, pages 127–135, 2012.
A. Bordes, X. Glorot, J. Weston, 和 Y. Bengio. 联合学习词汇和含义表示以进行开放文本语义解析。在 AISTATS 会议上，页码 127-135，2012 年。 -
[4]
A. Bordes, J. Weston, R. Collobert, and Y. Bengio.
Learning structured embeddings of knowledge bases.
In AAAI’11, 2011.
A. Bordes, J. Weston, R. Collobert, 和 Y. Bengio. 学习知识库的结构化嵌入。在 AAAI’11, 2011 年。 -
[5]
R. Borges, A. d’Avila Garcez, and L.C. Lamb.
Learning and representing temporal knowledge in recurrent networks.
IEEE Trans. Neural Networks, 22(12):2409–2421, 2011.
R. Borges, A. d’Avila Garcez, 和 L.C. Lamb. 在循环网络中学习和表示时间知识. IEEE 神经网络学报, 22(12):2409–2421, 2011. -
[6]
William W. Cohen, Fan Yang, and Kathryn Mazaitis.
Tensorlog: Deep learning meets probabilistic dbs.
CoRR, abs/1707.05390, 2017.
William W. Cohen，Fan Yang 和 Kathryn Mazaitis。Tensorlog：深度学习遇见概率数据库。CoRR，abs/1707.05390，2017。 -
[7]
A. d’Avila Garcez and G. Zaverucha.
The connectionist inductive learning and logic programming system.
Appl. Intelligence, 11(1):59–77, 1999.
A. d'Avila Garcez 和 G. Zaverucha。 连接主义归纳学习和逻辑编程系统。 应用智能，11(1)：59-77，1999 年。 -
[8]
A.S. d’Avila Garcez and L.C. Lamb.
Reasoning about time and knowledge in neural symbolic learning
systems.
In NIPS, pages 921–928, 2003.
A.S. d'Avila Garcez 和 L.C. Lamb。神经符号学习系统中关于时间和知识的推理。在 NIPS 中，第 921-928 页，2003 年。 -
[9]
A.S. d’Avila Garcez and L.C. Lamb.
A connectionist computational model for epistemic and temporal
reasoning.
Neur. Computation, 18(7):1711–1738, 2006.
A.S. d'Avila Garcez 和 L.C. Lamb。认知和时间推理的连接主义计算模型。神经计算，18(7)：1711-1738，2006 年。 -
[10]
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen.
Problog: A probabilistic prolog and its application in link
discovery.
In Proceedings of the 20th International Joint Conference on
Artifical Intelligence, IJCAI’07, pages 2468–2473, San Francisco, CA, USA,
2007. Morgan Kaufmann Publishers Inc.
Luc De Raedt，Angelika Kimmig 和 Hannu Toivonen。Problog：一种概率 Prolog 及其在链接发现中的应用。在第 20 届国际人工智能联合会议论文集中，IJCAI’07，页码 2468-2473，2007 年，美国加利福尼亚州旧金山。Morgan Kaufmann Publishers Inc. -
[11]
Frank DiMaio and Jude Shavlik.
Learning an approximation to inductive logic programming clause
evaluation.
In Rui Camacho, Ross King, and Ashwin Srinivasan, editors, Inductive Logic Programming, pages 80–97, Berlin, Heidelberg, 2004.
Springer Berlin Heidelberg.
Frank DiMaio 和 Jude Shavlik。学习归纳逻辑编程子句评估的近似值。在 Rui Camacho，Ross King 和 Ashwin Srinivasan 编辑的《归纳逻辑编程》中，第 80-97 页，2004 年，柏林，海德堡，Springer 柏林海德堡。 -
[12]
I. Donadello, L. Serafini, and A. S. d’Avila Garcez.
Logic tensor networks for semantic image interpretation.
In IJCAI-17, pages 1596–1602, 2017.
I. Donadello, L. Serafini, 和 A. S. d’Avila Garcez. 逻辑张量网络用于语义图像解释. 在 IJCAI-17, 页码 1596–1602, 2017. -
[13]
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou.
Neural logic machines.
2019.
洪华东，毛佳源，林田，王冲，李丽红和周丹尼。神经逻辑机。2019。 -
[14]
R. Evans and E. Grefenstette.
Learning explanatory rules from noisy data.
JAIR, 61:1–64, 2018.
R. Evans 和 E. Grefenstette. 从嘈杂数据中学习解释性规则. JAIR, 61:1–64, 2018. -
[15]
M. França, G. Zaverucha, and A. Garcez.
Fast relational learning using bottom clause propositionalization
with artificial neural networks.
Mach. Learning, 94(1):81–104, 2014.
M. França, G. Zaverucha 和 A. Garcez. 使用人工神经网络进行底层子句命题化的快速关系学习。机器学习，94(1)：81–104，2014 年。 -
[16]
A. d’Avila Garcez, L.C. Lamb, and D.M. Gabbay.
Neural-Symbolic Cognitive Reasoning.
Springer, 2009.
A. d'Avila Garcez, L.C. Lamb, and D.M. Gabbay. 神经符号认知推理. Springer, 2009. -
[17]
A.S. d’Avila Garcez, D. Gabbay, and K.. Broda.
Neural-Symbolic Learning System: Foundations and Applications.
Springer, 2002.
A.S. d'Avila Garcez, D. Gabbay, 和 K. Broda. 神经符号学习系统：基础与应用. Springer, 2002. -
[18]
Ross Girshick.
Fast r-cnn.
In Proceedings of the 2015 IEEE International Conference on
Computer Vision (ICCV), ICCV ’15, pages 1440–1448, Washington, DC, USA,
2015. IEEE Computer Society.
Ross Girshick. Fast r-cnn. 在 2015 年 IEEE 国际计算机视觉大会(ICCV)论文集中，ICCV ’15，第 1440-1448 页，2015 年，美国华盛顿特区。IEEE 计算机学会。 -
[19]
Marco Gori.
Machine Learning: A Constraint-Based Approach.
Morgan Kaufmann, 2018.
马可·戈里。机器学习：基于约束的方法。Morgan Kaufmann，2018 年。 -
[20]
Kalanit Grill-Spector and Rafael Malach.
The human visual cortex.
Annual Review of Neuroscience, 27(1):649–677, 2004.
Kalanit Grill-Spector 和 Rafael Malach。 人类视觉皮层。 神经科学年度评论，27（1）：649-677，2004 年。 -
[21]
B. Hammer and P. Hitzler, editors.
Perspectives of Neural-Symbolic Integration.
Springer, 2007.
B. Hammer 和 P. Hitzler，编辑。神经符号整合的视角。Springer，2007 年。 -
[22]
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt
Schiele, and Trevor Darrell.
Generating visual explanations.
In Computer Vision - ECCV 2016 - 14th European Conference,
pages 3–19, 2016.
Lisa Anne Hendricks，Zeynep Akata，Marcus Rohrbach，Jeff Donahue，Bernt Schiele 和 Trevor Darrell。生成视觉解释。在计算机视觉 - ECCV 2016 - 第 14 届欧洲会议，第 3-19 页，2016 年。 -
[23]
Z. Hu, X. Ma, Z. Liu, E. Hovy, and E. Xing.
Harnessing deep neural networks with logic rules.
In ACL, 2016.
Z. 胡，X. 马，Z. 刘，E. Hovy 和 E. Xing。利用逻辑规则驾驭深度神经网络。在 ACL，2016 年。 -
[24]
Henrik Jacobsson.
Rule extraction from recurrent neural networks: A taxonomy and
review.
Neural Comput., 17(6):1223–1263, June 2005.
Henrik Jacobsson. 从循环神经网络中提取规则：分类和综述。神经计算，17(6)：1223–1263，2005 年 6 月。 -
[25]
R. Khardon and D. Roth.
Learning to reason.
J. ACM, 44(5), 1997.
R. Khardon 和 D. Roth. 学习推理. J. ACM, 44(5), 1997. -
[26]
L.C. Lamb, R.V. Borges, and A.S. d’Avila Garcez.
A connectionist cognitive model for temporal synchronisation and
learning.
In AAAI, pages 827–832, 2007.
L.C. Lamb, R.V. Borges, 和 A.S. d’Avila Garcez. 一个用于时间同步和学习的连接主义认知模型. 在 AAAI 会议上, 页码 827–832, 2007. -
[27]
Y. LeCun, Y. Bengio, and G. Hinton.
Deep learning.
Nature, 521(7553):436–444, 2015.
Y. LeCun, Y. Bengio, 和 G. Hinton. 深度学习. 自然, 521(7553):436–444, 2015. -
[28]
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc
De Raedt.
Deepproblog: Neural probabilistic logic programming.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi,
and R. Garnett, editors, Advances in Neural Information Processing
Systems 31, pages 3749–3759. Curran Associates, Inc., 2018.
Robin Manhaeve，Sebastijan Dumancic，Angelika Kimmig，Thomas Demeester 和 Luc De Raedt。Deepproblog：神经概率逻辑编程。在 S. Bengio，H. Wallach，H. Larochelle，K. Grauman，N. Cesa-Bianchi 和 R. Garnett 编辑，Advances in Neural Information Processing Systems 31，页 3749-3759。Curran Associates，Inc.，2018。 -
[29]
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and
Sentences From Natural Supervision.
In International Conference on Learning Representations, 2019.
毛佳源，甘创，Pushmeet Kohli，Joshua B. Tenenbaum 和吴家俊。《神经符号概念学习者：从自然监督中解释场景、单词和句子》。2019 年国际学习表示会议。 -
[30]
Stephen Muggleton.
Inverse entailment and progol.
New Generation Computing, 13(3):245–286, Dec 1995.
Stephen Muggleton。逆向蕴涵和 Progol。新一代计算，13(3)：245–286，1995 年 12 月。 -
[31]
Stephen Muggleton.
Stochastic logic programs.
In New Generation Computing. Academic Press, 1996.
Stephen Muggleton. 随机逻辑程序。在新一代计算中。学术出版社，1996 年。 -
[32]
H. Palangi, P. Smolensky, X. He, and L. Deng.
Question-answering with grammatically-interpretable representations.
In AAAI, 2018.
H. Palangi, P. Smolensky, X. He, 和 L. Deng. 具有语法可解释表示的问答。在 AAAI，2018 年。 -
[33]
E. Parisotto, A.-R. Mohamed, R. Singh, L. Li, D. Zhou, and P. Kohli.
Neuro-symbolic program synthesis.
In ICLR, 2017.
E. Parisotto, A.-R. Mohamed, R. Singh, L. Li, D. Zhou, 和 P. Kohli. 神经符号程序合成. 在 ICLR, 2017. -
[34]
L. de Penning, A. d’Avila Garcez, L.C. Lamb, and J-J. Meyer.
A neural-symbolic cognitive agent for online learning and reasoning.
In IJCAI, pages 1653–1658, 2011.
L. de Penning, A. d’Avila Garcez, L.C. Lamb, 和 J-J. Meyer. 一个神经符号认知代理人用于在线学习和推理。在 IJCAI，页码 1653-1658，2011 年。 -
[35]
G. Pinkas.
Reasoning, nonmonotonicity and learning in connectionist networks
that capture propositional knowledge.
Artif. Intell., 77(2):203–247, 1995.
G. Pinkas. 捕捉命题知识的连接主义网络中的推理、非单调性和学习。人工智能，77(2)：203–247，1995 年。 -
[36]
Cristiano Grijó Pitangui and Gerson Zaverucha.
Learning theories using estimation distribution algorithms and
(reduced) bottom clauses.
In Stephen H. Muggleton, Alireza Tamaddoni-Nezhad, and Francesca A.
Lisi, editors, Inductive Logic Programming, pages 286–301, Berlin,
Heidelberg, 2012. Springer Berlin Heidelberg.
Cristiano Grijó Pitangui 和 Gerson Zaverucha。使用估计分布算法和（简化的）底部子句学习理论。在 Stephen H. Muggleton，Alireza Tamaddoni-Nezhad 和 Francesca A. Lisi 编辑的《归纳逻辑编程》中，第 286-301 页，2012 年，柏林，海德堡，Springer 柏林海德堡。 -
[37]
Tomaso A. Poggio and Fabio Anselmi.
Visual Cortex and Deep Networks: Learning Invariant
Representations.
The MIT Press, 1st edition, 2016.
Tomaso A. Poggio 和 Fabio Anselmi。视觉皮层和深度网络：学习不变表示。麻省理工学院出版社，第 1 版，2016 年。 -
[38]
H. Poon and P. Domingos.
Sum-product networks: A new deep architecture.
In 2011 IEEE Int. Conf. on Computer Vision Workshops
(ICCV Workshops), 2011.
H. Poon 和 P. Domingos。Sum-product networks: 一种新的深度架构。在 2011 年 IEEE 计算机视觉研讨会（ICCV Workshops）中。2011。 -
[39]
Tim Rocktäschel and Sebastian Riedel.
Learning knowledge base inference with neural theorem provers.
In Proceedings of the 5th Workshop on Automated Knowledge Base
Construction, pages 45–50, San Diego, CA, June 2016. Association for
Computational Linguistics.
Tim Rocktäschel 和 Sebastian Riedel。使用神经定理证明器学习知识库推理。在第 5 届自动知识库构建研讨会论文集中，第 45-50 页，2016 年 6 月，加利福尼亚州圣迭戈。计算语言学协会。 -
[40]
S.J. Russell, S. Hauert, R. Altman, and M. Veloso.
Ethics of artificial intelligence: Four leading researchers share
their concerns and solutions for reducing societal risks from intelligent
machines.
Nature, 521:415–418, 2015.
S.J. Russell, S. Hauert, R. Altman, 和 M. Veloso. 人工智能伦理学：四位领先研究人员分享他们对减少智能机器对社会风险的关注和解决方案。自然杂志，521:415–418，2015 年。 -
[41]
A. Santoro, D. Raposo, D. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and
T. Lillicrap.
A simple neural network module for relational reasoning.
In NIPS, 2017.
A. Santoro, D. Raposo, D. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. 一个用于关系推理的简单神经网络模块。在 NIPS，2017 年。 -
[42]
Luciano Serafini and Artur S. d’Avila Garcez.
Learning and reasoning with logic tensor networks.
In AI*IA, pages 334–348, 2016.
Luciano Serafini 和 Artur S. d'Avila Garcez。使用逻辑张量网络进行学习和推理。在 AI*IA 中，第 334-348 页，2016 年。 -
[43]
Ehsan Shokri-Kojori, Michael A. Motes, Bart Rypma, and Daniel C. Krawczyk.
The network architecture of cortical processing in visuo-spatial
reasoning.
Scientific Reports, 2, 2012.
Ehsan Shokri-Kojori，Michael A. Motes，Bart Rypma 和 Daniel C. Krawczyk。视觉空间推理皮层处理的网络架构。科学报告，2012 年，2。 -
[44]
Z. Si and S. C. Zhu.
Learning and-or templates for object recognition and detection.
IEEE Trans. Pattern Analysis and Mach. Intell.,
35(9):2189–2205, Sept 2013.
Z. Si 和 S. C. Zhu。用于目标识别和检测的学习和-或模板。IEEE 模式分析和机器智能交易，35(9)：2189-2205，2013 年 9 月。 -
[45]
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez,
T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui,
L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis.
Mastering the game of go without human knowledge.
Nature, 550(354), 2017.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. 不借助人类知识掌握围棋游戏。《自然》杂志，第 550 卷(354 页)，2017 年。 -
[46]
P. Smolensky.
Constituent structure and explanation in an integrated
connectionist/symbolic cognitive architecture.
In Connectionism: Debates on Psychological Explanation. 1995.
P. Smolensky. 组成结构与整合连接主义/符号认知架构中的解释。在《连接主义：心理解释的辩论》中。1995 年。 -
[47]
R. Socher, D. Chen, C. Manning, and A. Ng.
Reasoning with neural tensor networks for knowledge base completion.
In NIPS, pages 926–934. 2013.
R. Socher, D. Chen, C. Manning, 和 A. Ng. 使用神经张量网络进行知识库补全的推理。在 NIPS，第 926-934 页。2013。 -
[48]
I. Sutskever and G. Hinton.
Using matrices to model symbolic relationship.
In NIPS. 2009.
I. Sutskever 和 G. Hinton。使用矩阵来建模符号关系。在 NIPS。2009 年。 -
[49]
G. Towell and J. Shavlik.
Knowledge-based artificial neural networks.
Artif. Intel., 70:119–165, 1994.
G. Towell 和 J. Shavlik. 基于知识的人工神经网络. 人工智能, 70:119–165, 1994. -
[50]
S. Tran and A. d’Avila Garcez.
Knowledge extraction from deep belief networks for images.
In IJCAI-2013 Workshop on Neural-Symbolic Learning and
Reasoning, 2013.
S. Tran 和 A. d'Avila Garcez. 从深度信念网络中提取图像知识。在 2013 年 IJCAI 神经符号学习与推理研讨会上。 -
[51]
S. Tran and A. Garcez.
Deep logic networks: Inserting and extracting knowledge from deep
belief networks.
IEEE T. Neur. Net. Learning Syst., (29):246–258, 2018.
S. Tran 和 A. Garcez。深度逻辑网络：从深度信念网络中插入和提取知识。IEEE T. Neur. Net. Learning Syst.，(29)：246-258，2018。 -
[52]
Son N. Tran.
Propositional knowledge representation and reasoning in restricted
boltzmann machines.
CoRR, abs/1705.10899, 2018.
Son N. Tran. 限制玻尔兹曼机中的命题知识表示与推理. CoRR, abs/1705.10899, 2018. -
[53]
L. Valiant.
Knowledge infusion.
In AAAI, 2006.
L. Valiant. 知识注入. 在 AAAI，2006 年。 -
[54]
L.G. Valiant.
Three problems in computer science.
J. ACM, 50(1):96–99, 2003.
L.G. Valiant. 计算机科学中的三个问题. J. ACM, 50(1):96–99, 2003. -
[55]
Qinglong Wang, Kaixuan Zhang, Alexander G. Ororbia II, Xinyu Xing, Xue Liu, and
C. Lee Giles.
An empirical evaluation of rule extraction from recurrent neural
networks.
Neural Computation, 30(9):2568–2591, 2018.
王庆龙，张凯旋，亚历山大·G·奥罗比亚二世，邢新宇，刘雪，以及 C.李·吉尔斯。从循环神经网络中提取规则的实证评估。《神经计算》，30(9)：2568-2591，2018。 -
[56]
Carter Wendelken and Lokendra Shastri.
Multiple instantiation and rule mediation in SHRUTI.
Connect. Sci., 16(3):211–217, 2004.
卡特·温德尔肯（Carter Wendelken）和洛肯德拉·沙斯特里（Lokendra Shastri）。SHRUTI 中的多重实例化和规则调解。Connect. Sci.，16（3）：211-217，2004 年。 -
[57]
Fan Yang, Zhilin Yang, and William W Cohen.
Differentiable learning of logical rules for knowledge base
reasoning.
In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information
Processing Systems 30, pages 2319–2328. Curran Associates, Inc., 2017.
Fan Yang，Zhilin Yang 和 William W Cohen。可微学习知识库推理的逻辑规则。在 I. Guyon，U. V. Luxburg，S. Bengio，H. Wallach，R. Fergus，S. Vishwanathan 和 R. Garnett 编辑的《神经信息处理系统 30》中，第 2319-2328 页。Curran Associates，Inc.，2017。 -
[58]
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh
Tenenbaum.
Neural-symbolic vqa: Disentangling reasoning from vision and language
understanding.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi,
and R. Garnett, editors, Advances in Neural Information Processing
Systems 31, pages 1031–1042. Curran Associates, Inc., 2018.
易可欣，吴佳俊，甘闯，安东尼奥·托拉尔巴，普什米特·科利和乔什·特南鲍姆。神经符号 VQA：从视觉和语言理解中分离推理。在 S. Bengio，H. Wallach，H. Larochelle，K. Grauman，N. Cesa-Bianchi 和 R. Garnett 编辑，神经信息处理系统 31 的进展，第 1031-1042 页。Curran Associates，Inc.，2018。
