Logic Tensor Networks for
Semantic Image Interpretation
逻辑张量网络用于语义图像解释

Semantic Image Interpretation (SII) is the task of generating a structured semantic description of the content of an image. This structured description can be represented as a labelled directed graph, where each vertex corresponds to a bounding box of an object in the image, and each edge represents a relation between pairs of objects; verteces are labelled with a set of object types and edges are labelled by the binary relations. Such a graph is also called a scene graph in [15].
语义图像解释（SII）是生成图像内容的结构化语义描述的任务。这种结构化描述可以表示为一个带标签的有向图，其中每个顶点对应于图像中的一个物体的边界框，每条边表示物体对之间的关系；顶点用一组物体类型标记，边由二元关系标记。这样的图在文献[15]中也被称为场景图。

A major obstacle to be overcome by SII is the so-called semantic gap [19], that is, the lack of a direct correspondence between low-level features of the image and high-level semantic descriptions. To tackle this problem, a system for SII must learn the latent correlations that may exist between the numerical features that can be observed in an image and the semantic concepts associated with the objects. It is in this learning process that the availability of relational background knowledge can be of great help. Thus, recent SII systems have sought to combine, or even integrate, visual features obtained from data and symbolic knowledge in the form of logical axioms [30, 4, 8].
SII 需要克服的一个主要障碍是所谓的语义鸿沟[19]，即图像的低级特征与高级语义描述之间缺乏直接对应。为了解决这个问题，SII 系统必须学习可能存在的图像中可观察到的数值特征与与对象相关的语义概念之间的潜在相关性。在这个学习过程中，关系背景知识的可用性可以提供很大帮助。因此，最近的 SII 系统已经试图结合甚至整合从数据中获得的视觉特征和逻辑公理形式的符号知识[30, 4, 8]。

The area of Statistical Relational Learning (SRL), or Statistical Artificial Intelligence (StarAI), seeks to combine data-driven learning, in the presence of uncertainty, with symbolic knowledge [29, 2, 13, 7, 26, 23]. However, only very few SRL systems have been applied to SII tasks (c.f. Section 2) due to the high complexity associated with image learning. Most systems for solving SII tasks have been based, instead, on deep learning and neural network models. These, on the other hand, do not in general offer a well-founded way of learning from data in the presence of relational logical constraints, requiring the neural models to be highly engineered from scratch.
统计关系学习（SRL）或统计人工智能（StarAI）领域旨在将数据驱动学习与符号知识相结合，以应对不确定性[29, 2, 13, 7, 26, 23]。然而，由于图像学习的高复杂性，只有极少数 SRL 系统被应用于 SII 任务（参见第 2 节）。大多数解决 SII 任务的系统基于深度学习和神经网络模型。另一方面，这些模型通常不能提供一种基于数据学习的良好方法，以应对关系逻辑约束的存在，需要从头开始高度工程化神经模型。

In this paper, we develop and apply for the first time, the SRL framework called Logic Tensor Networks (LTNs) to computationally challenging SII tasks. LTNs combine learning in deep networks with relational logical constraints [27]. It uses a First-order Logic (FOL) syntax interpreted in the real numbers, which is implemented as a deep tensor network. Logical terms are interpreted as feature vectors in a real-valued $n$-dimensional space. Function symbols are interpreted as real-valued functions, and predicate symbols as fuzzy logic relations. This syntax and semantics, called real semantics, allow LTNs to learn efficiently in hybrid domains, where elements are composed of both numerical and relational information.
在本文中，我们首次开发并应用了名为逻辑张量网络（LTNs）的 SRL 框架，用于计算具有挑战性的 SII 任务。LTNs 将深度网络中的学习与关系逻辑约束相结合[27]。它使用在实数中解释的一阶逻辑（FOL）语法，这被实现为深度张量网络。逻辑术语被解释为实值 $n$ 维空间中的特征向量。函数符号被解释为实值函数，谓词符号被解释为模糊逻辑关系。这种称为实语义的语法和语义允许 LTNs 在混合域中高效学习，其中元素由数值和关系信息组成。

We argue, therefore, that LTNs are a good candidate for learning SII because they can express relational knowledge in FOL which serves as constraints on the data-driven learning within tensor networks. Being LTN a logic, it provides a notion of logical consequence, which forms the basis for learning within LTNs, which is defined as best satisfiability, c.f. Section 4. Solving the best satisfiability problem amounts to finding the latent correlations that may exist between a relational background knowledge and numerical data attributes. This formulation enables the specification of learning as reasoning, a unique characteristic of LTNs, which is seen as highly relevant for SII.
因此，我们认为 LTN 是学习 SII 的一个很好的候选者，因为它们可以在 FOL 中表达关系知识，这作为张量网络中数据驱动学习的约束。作为一种逻辑，LTN 提供了逻辑推论的概念，这构成了 LTN 学习的基础，被定义为最佳可满足性，参见第 4 节。解决最佳可满足性问题意味着找到可能存在于关系背景知识和数值数据属性之间的潜在相关性。这种表述使学习作为推理的规范化成为可能，这是 LTN 的一个独特特征，被认为对 SII 非常相关。

This paper specifies SII within LTNs, evaluating it on two important tasks: (i) the classification of bounding boxes, and (ii) the detection of the part-of relation between any two bounding boxes. Both tasks are evaluated using the PASCAL-Part  dataset [5]. It is shown that LTNs improve the performance of the state-of-the-art object classifier Fast R-CNN [11] on the bounding box classification task. LTNs also outperform a rule-based heuristic (which uses the inclusion ratio of two bounding boxes) in the detection of part-of relations between objects. Finally, LTNs are evaluated on their ability to handle errors, specifically misclassifications of objects and part-of relations. Very large visual recognition datasets now exist which are noisy [24], and it is important for learning systems to become robust to noise. LTNs were trained systematically on progressively noisier datasets, with results on both SII tasks showing that LTN’s logical constraints are capable of adding robustness to the system, in the presence of errors in the labels of the training data.
本文在 LTNs 中明确了 SII，并对其进行了评估，涉及两个重要任务：(i) 边界框的分类，以及 (ii) 任意两个边界框之间部分关系的检测。这两个任务使用 PASCAL-Part 数据集[5]进行评估。研究结果表明，LTNs 提高了最先进的目标分类器 Fast R-CNN[11]在边界框分类任务上的性能。LTNs 在检测对象之间部分关系方面也优于基于规则的启发式方法（使用两个边界框的包含比率）。最后，LTNs 在处理错误的能力上进行了评估，特别是对象和部分关系的错误分类。现在存在非常大的视觉识别数据集，其中存在噪声[24]，学习系统变得对噪声具有鲁棒性变得非常重要。LTNs 系统地在逐渐嘈杂的数据集上进行了训练，对 SII 任务的结果显示，LTN 的逻辑约束能够在训练数据的标签存在错误的情况下为系统增加鲁棒性。

The paper is organized as follows: Section 2 contrasts the LTN approach with related work which integrate visual features and background knowledge for SII. Section 3 specifies LTNs in the context of SII. Section 4 defines the best satisfiability problem in this context, which enables the use of LTNs for SII. Section 5 describes in detail the comparative evaluations of LTNs on the SII tasks. Section 6 concludes the paper and discusses directions for future work.
本文的组织如下：第 2 节将 LTN 方法与整合视觉特征和背景知识用于 SII 的相关工作进行对比。第 3 节在 SII 的背景下具体说明了 LTNs。第 4 节定义了这一背景下的最佳可满足性问题，从而实现了 LTNs 在 SII 中的应用。第 5 节详细描述了 LTNs 在 SII 任务上的比较评估。第 6 节总结了本文并讨论了未来工作的方向。

The idea of exploiting logical background knowledge to improve SII tasks dates back to the early days of AI. In what follows, we review the most recent results in the area in comparison with LTNs.
利用逻辑背景知识来改进 SII 任务的想法可以追溯到人工智能的早期。接下来，我们将与 LTNs 进行比较，回顾该领域最新的研究结果。

Logic-based approaches have used Description Logics (DL), where the basic components of the scene are all assumed to have been already discovered (e.g. simple object types or spatial relations). Then, with logical reasoning, new facts can be derived in the scene from these basic components [19, 21]. Other logic-based approaches have used fuzzy DL to tackle uncertainty in the basic components [14, 6, 1]. These approaches have limited themselves to spatial relations or to refining the labels of the objects detected. In [8], the scene interpretation is created by combining image features with constraints defined using DL, but the method is tailored to the part-of relation and cannot be extended easily to account for other relations. LTNs, on the other hand, should be able to handle any semantic relation. In [18, 10], a symbolic Knowledge-base is used to improve object detection, but only the subsumption relation is explored and it is not possible to inject more complex knowledge using logical axioms.
基于逻辑的方法已经使用了描述逻辑（DL），其中场景的基本组件都被假定已经被发现（例如简单的对象类型或空间关系）。然后，通过逻辑推理，可以从这些基本组件中推导出场景中的新事实。其他基于逻辑的方法使用模糊 DL 来处理基本组件中的不确定性。这些方法局限于空间关系或者对检测到的对象的标签进行细化。在[8]中，通过将图像特征与使用 DL 定义的约束相结合来创建场景解释，但该方法专门针对部分关系，不容易扩展到其他关系。另一方面，逻辑张量网络（LTNs）应该能够处理任何语义关系。在[18, 10]中，使用符号知识库来改进对象检测，但只探讨了包含关系，无法使用逻辑公理注入更复杂的知识。

A second group of approaches seeks to encode background knowledge and visual features within probabilistic graphical models. In [30, 20], visual features are combined with knowledge gathered from datasets, web resources or annotators, about object labels, properties such as shape, colour and size, and affordances, using Markov Logic Networks (MLNs) [25] to predict facts in unseen images. Due to the specific knowledge-base schema adopted, the effectiveness of MLNs in this domain is evaluated only for Horn clauses, although the language of MLNs is more general. As a result, it is not easy to evaluate how the approach may perform with more complex axioms. In [2], a probabilistic fuzzy logic is used, but not with real semantics. Clauses are weighted and universally-quantified formulas are instantiated, as done by MLNs. This is different from LTNs where the universally-quantified formulas are computed by using an aggregation operation, which avoids the need for instantiating all variables.
一组方法试图在概率图模型中编码背景知识和视觉特征。在[30, 20]中，视觉特征与从数据集、网络资源或注释者收集的关于对象标签、形状、颜色和大小等属性以及功能的知识相结合，使用马尔可夫逻辑网络（MLNs）[25]来预测未见图像中的事实。由于采用了特定的知识库模式，MLNs 在这个领域的有效性仅针对 Horn 子句进行评估，尽管 MLNs 的语言更通用。因此，很难评估这种方法在处理更复杂公理时的表现。在[2]中，使用了概率模糊逻辑，但没有真实语义。子句被加权，普遍量化的公式被实例化，就像 MLNs 所做的那样。这与 LTNs 不同，LTNs 中的普遍量化公式是通过使用聚合操作计算的，这避免了需要实例化所有变量。

In other related work, [4, 16] encode background knowledge into a generic Conditional Random Field (CRF), where the nodes represent detected objects and the edges represent logical relationships between objects. The task is to find a correct labelling for this graph. In [4], the edges encode logical constraints on a knowledge-base specified in DL. Although these ideas are close in spirit to the approach presented in this paper, they are not formalised as in LTNs, which use a deep tensor network and first-order logic, rather than CRFs or DL. In general, the logical theory behind the functions to be defined in the CRF is unclear. In [16], potential functions are defined as text priors such as co-occurrence of terms found in the image descriptions of Flickr.
在其他相关工作中，[4, 16] 将背景知识编码到一个通用的条件随机场（CRF）中，其中节点代表检测到的对象，边代表对象之间的逻辑关系。任务是为这个图找到正确的标记。在[4]中，边编码了在 DL 中指定的知识库上的逻辑约束。尽管这些想法在精神上与本文提出的方法接近，但它们没有像 LTNs 那样形式化，LTNs 使用深度张量网络和一阶逻辑，而不是 CRFs 或 DL。一般来说，CRF 中要定义的函数背后的逻辑理论是不清楚的。在[16]中，潜在函数被定义为文本先验，例如在 Flickr 的图像描述中发现的术语共现。

In a final group of approaches, here called language-priors, background knowledge is taken from linguistic models [22, 17]. In [22], a neural network is built integrating visual features and a linguistic model to predict semantic relationships between bounding boxes. The linguistic model is a set of rules derived from WordNet [9], stating which types of semantic relationships occur between a subject and an object. In [17], a similar neural network is proposed for the same task but with a more sophisticated language model, embedding in the same vector space triples of the form subject-relation-object, such that semantically similar triples are mapped closely together in the embedding space. In this way, even if no examples exist of some triples in the data, the relations can be inferred from similarity to more frequent triples. A drawback, however, is the possibility of inferring inconsistent triples, such as e.g. man-eats-chair, due to the embedding. LTNs avoid this problem with a logic-based approach (in the above example, with an axiom to the effect that chairs are not normally edible). LTNs can also handle exceptions, offering a system capable of dealing with crisp axioms and real-valued data, as specified in what follows.
在最后一组方法中，这里称为语言先验，背景知识来自语言模型[22, 17]。在[22]中，构建了一个神经网络，集成了视觉特征和语言模型，以预测边界框之间的语义关系。语言模型是从 WordNet[9]中导出的一组规则，说明主语和宾语之间发生哪些类型的语义关系。在[17]中，提出了一个类似的神经网络用于相同的任务，但使用了更复杂的语言模型，在同一个向量空间中嵌入了主题-关系-客体形式的三元组，使语义上相似的三元组在嵌入空间中紧密映射在一起。通过这种方式，即使数据中不存在某些三元组的示例，也可以从与更频繁的三元组相似性推断出关系。然而，一个缺点是可能从嵌入中推断出不一致的三元组，例如人-吃-椅子，LTNs 通过基于逻辑的方法避免了这个问题（在上面的例子中，通过一个关于椅子通常不可食用的公理）。 LTNs 也可以处理异常情况，提供一个能够处理清晰公理和实值数据的系统，如下所述。

Let $\mathcal{L}$ be a first-order logic language, whose signature is composed of three disjoint sets $\mathcal{C}$, $\mathcal{F}$ and $\mathcal{P}$, denoting constants, functions and predicate symbols, respectively. For any function or predicate symbol $s$, let $\alpha(s)$ denote its arity. Logical formulas in $\mathcal{L}$ allow one to specify relational knowledge, e.g. the atomic formula $\mathsf{partOf}(o_{1},o_{2})$, stating that object $o_{1}$ is a part of object $o_{2}$, the formulae $\forall xy(\mathsf{partOf}(x,y)\rightarrow\neg\mathsf{partOf}(y,x))$, stating that the relation $\mathsf{partOf}$ is asymmetric, or $\forall x(\mathsf{Cat}(x)\rightarrow\exists y(\mathsf{partOf}(x,y)\wedge\mathsf{Tail}(y)))$, stating that every cat should have a tail. In addition, exceptions are handled by allowing formulas to be interpreted in fuzzy logic, such that in the presence of an example of, say, a tailless cat, the above formula can be interpreted naturally as normally, every cat has a tail; this will be exemplified later.
令 $\mathcal{L}$ 为一个一阶逻辑语言，其签名由三个不相交的集合 $\mathcal{C}$ 、 $\mathcal{F}$ 和 $\mathcal{P}$ 组成，分别表示常量、函数和谓词符号。对于任何函数或谓词符号 $s$ ，令 $\alpha(s)$ 表示其元数。 $\mathcal{L}$ 中的逻辑公式允许指定关系知识，例如原子公式 $\mathsf{partOf}(o_{1},o_{2})$ ，表示对象 $o_{1}$ 是对象 $o_{2}$ 的一部分，公式 $\forall xy(\mathsf{partOf}(x,y)\rightarrow\neg\mathsf{partOf}(y,x))$ ，表示关系 $\mathsf{partOf}$ 是非对称的，或 $\forall x(\mathsf{Cat}(x)\rightarrow\exists y(\mathsf{partOf}(x,y)\wedge\mathsf{Tail}(y)))$ ，表示每只猫都应该有尾巴。此外，通过允许公式在模糊逻辑中进行解释来处理异常情况，例如，在存在无尾巴的猫的情况下，上述公式可以自然地被解释为通常情况下每只猫都有尾巴；这将在后面举例说明。

Given a grounding $\mathcal{G}$, the semantics of closed terms and atomic formulas is defined as follows:
给定一个基础 $\mathcal{G}$ ，封闭项和原子公式的语义定义如下：

	$\displaystyle\mathcal{G}(f(t_{1},\dots,t_{m}))$	$\displaystyle=\mathcal{G}(f)(\mathcal{G}(t_{1}),\dots,\mathcal{G}(t_{m}))$
	$\displaystyle\mathcal{G}(P(t_{1},\dots,t_{m}))$	$\displaystyle=\mathcal{G}(P)(\mathcal{G}(t_{1}),\dots,\mathcal{G}(t_{m}))$

The LTN semantics for $\forall$ is defined in [27] using the $\min$ operator, that is, $\mathcal{G}(\forall x\phi(x))=\min_{t\in\mathit{term}(\mathcal{L})}\mathcal{G}(\phi(t)))$, where $\mathit{term}(\mathcal{L})$ is the set of instantiated terms of $\mathcal{L}$. This, however, is inadequate for our purposes as it does not tolerate exceptions well (the presence of a single exception to the universally-quantified formulae, such as e.g. a cat without a tail, would falsify the formulae. Instead, our intention in SII is that the more examples there are that satisfy a formulae $\phi(x)$, the higher the truth-value of $\forall x\phi(x)$ should be. To capture this, we use for the semantics of $\forall$ a mean-operator, as follows:
LTN 语义学对 $\forall$ 的定义在[27]中使用 $\min$ 运算符定义，即 $\mathcal{G}(\forall x\phi(x))=\min_{t\in\mathit{term}(\mathcal{L})}\mathcal{G}(\phi(t)))$ ，其中 $\mathit{term}(\mathcal{L})$ 是 $\mathcal{L}$ 的实例化术语集合。然而，这对于我们的目的是不够的，因为它不能很好地容忍异常情况（对于普遍量化的公式，如一个没有尾巴的猫等存在一个异常情况，将使公式变为假）。相反，在 SII 中，我们的意图是，满足公式 $\phi(x)$ 的示例越多， $\forall x\phi(x)$ 的真值就应该越高。为了捕捉这一点，我们使用均值运算符来定义 $\forall$ 的语义，如下所示：

$$\mathcal{G}(\forall x\phi(x))=\lim_{T\rightarrow\mathit{term}(\mathcal{L})}mean_{p}(\mathcal{G}(\phi(t))\mid t\in T)$$

where $mean_{p}(x_{1},\dots,x_{d})=\left(\frac{1}{d}\sum_{i=1}^{d}x_{i}^{p}\right)^{\frac{1}{p}}$ for $p\in\mathbb{Z}$. ³³3The popular mean operators, arithmetic, geometric and harmonic mean, are obtained by setting $p=1,2,$ and $-1$, respectively.
流行的平均运算符，算术平均数、几何平均数和调和平均数，分别通过设置 $p=1,2,$ 和 $-1$ 获得。
在 $p\in\mathbb{Z}$ 的情况下 $mean_{p}(x_{1},\dots,x_{d})=\left(\frac{1}{d}\sum_{i=1}^{d}x_{i}^{p}\right)^{\frac{1}{p}}$ 。 ³

Finally, the classical semantics of $\exists$ is uniquely determined by the semantics of $\forall$, by making $\exists$ equivalent to $\neg\forall\neg$. This approach, however, has a drawback too when it comes to SII: if we adopt, for instance, the arithmetic mean for the semantic of $\forall$ then $\mathcal{G}(\forall x\phi(x))=\mathcal{G}(\exists x\phi(x))$. Therefore, we shall interpret existential quantification via Skolemization: every formula of the form $\forall x_{1},\dots,x_{n}(\dots\exists y\phi(x_{1},\dots,x_{n},y))$ is rewritten as $\forall x_{1},\dots,x_{n}(\dots\phi(x_{1},\dots,x_{n},f(x_{1},\dots,x_{n})))$, by introducing a new $n$-ary function symbol, called Skolem function. In this way, existential quantifiers can be eliminated from the language by introducing Skolem functions.
最后， $\exists$ 的经典语义是通过使 $\exists$ 等价于 $\neg\forall\neg$ 来唯一确定 $\forall$ 的语义。然而，当涉及到 SII 时，这种方法也有一个缺点：如果我们采用例如算术平均值来解释 $\forall$ 的语义，那么 $\mathcal{G}(\forall x\phi(x))=\mathcal{G}(\exists x\phi(x))$ 。因此，我们将通过斯科伦化来解释存在量词：形如 $\forall x_{1},\dots,x_{n}(\dots\exists y\phi(x_{1},\dots,x_{n},y))$ 的每个公式都被重写为 $\forall x_{1},\dots,x_{n}(\dots\phi(x_{1},\dots,x_{n},f(x_{1},\dots,x_{n})))$ ，引入一个新的 $n$ 元函数符号，称为斯科伦函数。通过引入斯科伦函数，存在量词可以从语言中消除。

A grounding for $\Sigma_{\text{SII}}$ can be defined as follows: each constant $b$, denoting a bounding box, can be associated with a set of geometric features and a set of semantic features obtained from the output of a bounding box detector. Specifically, each bounding box is associated with geometric features describing the position and the dimension of the bounding box, and semantic features describing the classification score returned by the bounding box detector for each class. For example, for each bounding box $b\in\mathcal{C}$, $C_{i}\in\mathcal{P}_{1}$, $\mathcal{G}(b)$ is the $\mathbb{R}^{4+|\mathcal{P}_{1}|}$ vector:
一个 $\Sigma_{\text{SII}}$ 的基础可以定义如下：每个常量 $b$ ，表示一个边界框，可以与一组几何特征和一组从边界框检测器的输出中获得的语义特征相关联。具体来说，每个边界框都与描述边界框位置和尺寸的几何特征以及描述边界框检测器为每个类别返回的分类分数的语义特征相关联。例如，对于每个边界框 $b\in\mathcal{C}$ ， $C_{i}\in\mathcal{P}_{1}$ ， $\mathcal{G}(b)$ 是 $\mathbb{R}^{4+|\mathcal{P}_{1}|}$ 向量：

$$\langle class(C_{1},b),\dots,class(C_{|\mathcal{P}_{1}|},b),x_{0}(b),y_{0}(b),x_{1}(b),y_{1}(b)\rangle$$

where the last four elements are the coordinates of the top-left and bottom-right corners of $b$, and $class(C_{i},b)\in[0,1]$ is the classification score of the bounding box detector for $b$.
最后四个元素是 $b$ 左上角和右下角的坐标， $class(C_{i},b)\in[0,1]$ 是边界框检测器对 $b$ 的分类分数。

An example of groundings for predicates can be defined by taking a one-vs-all multi-classifier approach, as follows. First, define the following grounding for each class $C_{i}\in\mathcal{P}_{1}$ (below, $\mathbf{x}=\left<x_{1},\dots,x_{|\mathcal{P}_{1}|+4}\right>$ is the vector corresponding to the grounding of a bounding box):
谓词的基础示例可以通过采用一对多多分类器方法来定义，具体如下。首先，为每个类别定义以下基础 $C_{i}\in\mathcal{P}_{1}$ （下面， $\mathbf{x}=\left<x_{1},\dots,x_{|\mathcal{P}_{1}|+4}\right>$ 是对边界框基础的向量）。

$\displaystyle\mathcal{G}(C_{i})(\mathbf{x})=\left\{\begin{array}[]{ll}1&\mbox{if }i=\mathop{\mathrm{argmax}}_{1\leq l\leq|\mathcal{P}_{1}|}x_{l}\\ 0&\mbox{otherwise}\end{array}\right.$

(3)

Then, a simple rule-based approach for defining a grounding for the $\mathsf{partOf}$ relation is based on the naïve assumption that the more a bounding box $b$ is contained within a bounding box $b^{\prime}$, the higher the probability should be that $b$ is part of $b^{\prime}$. Accordingly, one can define $\mathcal{G}(\mathsf{partOf}(b,b^{\prime}))$ as the inclusion ratio $\mathit{ir}(b,b^{\prime})$ of bounding box $b$, with grounding $\mathbf{x}$, into bounding box $b^{\prime}$, with grounding $\mathbf{x}^{\prime}$ (formally, $\mathit{ir}(b,b^{\prime})=\frac{area(b\cap b^{\prime})}{area(b)}$). A slightly more sophisticated rule-based grounding for $\mathsf{partOf}$ (used as baseline in the experiments to follow) takes into account also type compatibilities by multiplying the inclusion ratio by a factor $w_{ij}$. Hence, we define $\mathcal{G}(\mathsf{partOf}(b,b^{\prime}))$ as follows:
然后，一个简单的基于规则的方法来定义 $\mathsf{partOf}$ 关系的基础是基于天真的假设，即边界框 $b$ 越多地包含在边界框 $b^{\prime}$ 中， $b$ 是 $b^{\prime}$ 的一部分的概率就应该越高。因此，可以将 $\mathcal{G}(\mathsf{partOf}(b,b^{\prime}))$ 定义为边界框 $b$ 的包含比率 $\mathit{ir}(b,b^{\prime})$ ，带有基础 $\mathbf{x}$ ，进入边界框 $b^{\prime}$ ，带有基础 $\mathbf{x}^{\prime}$ （形式上， $\mathit{ir}(b,b^{\prime})=\frac{area(b\cap b^{\prime})}{area(b)}$ ）。一个稍微更复杂的基于规则的 $\mathsf{partOf}$ 基础（在接下来的实验中用作基准）还考虑了类型的兼容性，通过将包含比率乘以一个因子 $w_{ij}$ 。因此，我们定义 $\mathcal{G}(\mathsf{partOf}(b,b^{\prime}))$ 如下：

$\displaystyle\left\{\begin{array}[]{ll}1&\mbox{if }\mathit{ir}(b,b^{\prime})\cdot\max_{ij=1}^{|\mathcal{P}_{1}|}(w_{ij}\cdot x_{i}\cdot x^{\prime}_{j})\geq th_{ir}\\ 0&\mbox{otherwise}\end{array}\right.$

(6)

for some threshold $th_{ir}$ (we use $th_{ir}>0.5$), and with $w_{ij}=1$ if $C_{i}$ is a part of $C_{j}$, and $0$ otherwise. Given the above grounding, we can compute the grounding of any atomic formula, e.g. $\mathsf{Cat}(b_{1})$, $\mathsf{Dog}(b_{2})$, $\mathsf{leg}(b_{3})$, $\mathsf{partOf}(b_{3},b_{1})$, $\mathsf{partOf}(b_{3},b_{2})$, thus expressing the degree of truth of the formula. The rule-based groundings (Eqs. (3) and (6)) may not satisfy some of the constraints to be imposed. For example, the classification score may be wrong, a bounding box may include another which is not in the part-of relation, etc. Furthermore, in many situations, it is not possible to define a grounding a priori. Instead, groundings may need to be learned automatically from examples, by optimizing the truth-values of the formulas in the background knowledge. This is discussed next.
对于某个阈值 $th_{ir}$ （我们使用 $th_{ir}>0.5$ ），如果 $C_{i}$ 是 $C_{j}$ 的一部分，则为 $w_{ij}=1$ ，否则为 $0$ 。在上述基础上，我们可以计算任何原子公式的基础，例如 $\mathsf{Cat}(b_{1})$ ， $\mathsf{Dog}(b_{2})$ ， $\mathsf{leg}(b_{3})$ ， $\mathsf{partOf}(b_{3},b_{1})$ ， $\mathsf{partOf}(b_{3},b_{2})$ ，从而表达公式的真实程度。基于规则的基础（方程（3）和（6））可能无法满足一些要求的约束条件。例如，分类得分可能是错误的，边界框可能包含另一个不在部分关系中的边界框等。此外，在许多情况下，不可能事先定义一个基础。相反，基础可能需要从示例中自动学习，通过优化背景知识中公式的真值。接下来将讨论这一点。

A partial grounding, denoted by $\hat{\mathcal{G}}$, is a grounding that is defined on a subset of the signature of $\mathcal{L}$. A grounding $\mathcal{G}$ is said to be a completion of $\hat{\mathcal{G}}$, if $\mathcal{G}$ is a grounding for $\mathcal{L}$ and coincides with $\hat{\mathcal{G}}$ on the symbols where $\hat{\mathcal{G}}$ is defined.
部分接地，表示为 $\hat{\mathcal{G}}$ ，是在 $\mathcal{L}$ 的签名的子集上定义的接地。如果 $\mathcal{G}$ 是 $\hat{\mathcal{G}}$ 的一个完成，那么如果 $\mathcal{G}$ 是 $\mathcal{L}$ 的一个接地并且在 $\hat{\mathcal{G}}$ 定义的符号上与 $\hat{\mathcal{G}}$ 相符，则称 $\mathcal{G}$ 是 $\hat{\mathcal{G}}$ 的一个完成。

According to the previous definition, deciding the satisfiability of $\langle\mathcal{K},\hat{\mathcal{G}}\rangle$ amounts to searching for a grounding $\hat{\mathcal{G}}$ such that all the formulas of $\mathcal{K}$ are mapped to 1. Differently from the classical satisfiability, when a GT is not satisfiable, we are interested in the best possible satisfaction that we can reach with a grounding. This is defined as follows.
根据先前的定义，决定 $\langle\mathcal{K},\hat{\mathcal{G}}\rangle$ 的可满足性相当于搜索一个基础 $\hat{\mathcal{G}}$ ，使得 $\mathcal{K}$ 的所有公式都映射到 1。与经典可满足性不同的是，当一个 GT 不可满足时，我们对于通过基础达到的最佳满足度感兴趣。定义如下。

Grounding $\mathcal{G}^{*}$ captures the latent correlation between the quantitative attribute of objects and their categorical and relational properties. Not all functions are suitable as a grounding; they should preserve some form of regularity. If $\mathcal{G}(\mathsf{Cat})(\mathbf{x})\approx 1$ (the bounding box with feature vector $\mathbf{x}$ contains a cat) then for every $\mathbf{x}^{\prime}$ close to $\mathbf{x}$ (i.e. for every bounding box with features similar to $\mathbf{x}$), one should have $\mathcal{G}(\mathsf{Cat})(\mathbf{x}^{\prime})\approx 1$. In particular, we consider groundings of the following form:
基础 $\mathcal{G}^{*}$ 捕捉了物体的定量属性与它们的分类和关系属性之间的潜在相关性。并非所有函数都适合作为基础；它们应该保留某种形式的规律性。如果 $\mathcal{G}(\mathsf{Cat})(\mathbf{x})\approx 1$ （具有特征向量 $\mathbf{x}$ 的边界框包含一只猫），那么对于每个接近 $\mathbf{x}$ 的 $\mathbf{x}^{\prime}$ （即对于每个具有类似 $\mathbf{x}$ 特征的边界框），应该有 $\mathcal{G}(\mathsf{Cat})(\mathbf{x}^{\prime})\approx 1$ 。特别地，我们考虑以下形式的基础：

Function symbols are grounded to linear transformations. If $f$ is a $m$-ary function symbol, then $\mathcal{G}(f)$ is of the form:
函数符号被基于线性变换。如果 $f$ 是一个 $m$ 元函数符号，则 $\mathcal{G}(f)$ 的形式为：

$$\mathcal{G}(f)(\mathbf{v})=M_{f}\mathbf{v}+N_{f}$$

where $\mathbf{v}=\left<\mathbf{v}_{1}^{\intercal},\dots,\mathbf{v}_{m}^{\intercal}\right>^{\intercal}$ is the $mn$-ary vector obtained by concatenating each $\mathbf{v}_{i}$. The parameters for $\mathcal{G}(f)$ are the $n\times mn$ real matrix $M_{f}$ and the $n$-vector $N_{f}$.
其中 $\mathbf{v}=\left<\mathbf{v}_{1}^{\intercal},\dots,\mathbf{v}_{m}^{\intercal}\right>^{\intercal}$ 是通过连接每个 $\mathbf{v}_{i}$ 获得的 $mn$ -元向量。 $\mathcal{G}(f)$ 的参数是 $n\times mn$ 实矩阵 $M_{f}$ 和 $n$ -元向量 $N_{f}$ 。

The grounding of an $m$-ary predicate $P$, namely $\mathcal{G}(P)$, is defined as a generalization of the neural tensor network (which has been shown effective at knowledge completion in the presence of simple logical constraints [28]), as a function from $\mathbb{R}^{mn}$ to $[0,1]$, as follows:
$m$ 元谓词 $P$ 的基础，即 $\mathcal{G}(P)$ ，被定义为神经张量网络的一种泛化（已被证明在简单逻辑约束存在的情况下对知识完成有效[28]），作为从 $\mathbb{R}^{mn}$ 到 $[0,1]$ 的函数，如下所示：

$\displaystyle\mathcal{G}(P)(\mathbf{v})=\sigma\left(u^{\intercal}_{P}\tanh\left(\mathbf{v}^{\intercal}W_{P}^{[1:k]}\mathbf{v}+V_{P}\mathbf{v}+b_{P}\right)\right)$

(7)

with $\sigma$ the sigmoid function. The parameters for $P$ are: $W_{P}^{[1:k]}$, a 3-D tensor in $\mathbb{R}^{k\times mn\times mn}$, $V_{P}\in\mathbb{R}^{k\times mn}$, $b_{P}\in\mathbb{R}^{k}$ and $u_{P}\in\mathbb{R}^{k}$. This last parameter performs a linear combination of the quadratic features given by the tensor product. With this encoding, the grounding (i.e. truth-value) of a clause can be determined by a neural network which first computes the grounding of the literals contained in the clause, and then combines them using the specific t-norm.
使用 $\sigma$ sigmoid 函数。 $P$ 的参数为： $W_{P}^{[1:k]}$ ，一个在 $\mathbb{R}^{k\times mn\times mn}$ 中的 3-D 张量， $V_{P}\in\mathbb{R}^{k\times mn}$ ， $b_{P}\in\mathbb{R}^{k}$ 和 $u_{P}\in\mathbb{R}^{k}$ 。最后一个参数执行张量积给出的二次特征的线性组合。通过这种编码，子句的基础（即真值）可以通过神经网络来确定，该网络首先计算子句中包含的文字的基础，然后使用特定的 t-范数将它们组合起来。

In what follows, we describe how a suitable GT can be built for SII. Let $\mathit{Pics}^{t}\subseteq\mathit{Pics}$ be a set of bounding boxes of images correctly labelled with the classes that they belong to, and let each pair of bounding boxes be correctly labelled with the part-of relation. In machine learning terminology, $\mathit{Pics}^{t}$ is a training set without noise. In real semantics, a training set can be represented by a theory $\mathcal{T}_{\mathrm{expl}}=\langle\mathcal{K}_{\mathrm{expl}},\hat{\mathcal{G}}\rangle$, where $\mathcal{K}_{\mathrm{expl}}$ contains the set of closed literals $C_{i}(b)$ (resp. $\neg C_{i}(b)$) and $\mathsf{partOf}(b,b^{\prime})$ (resp. $\neg\mathsf{partOf}(b,b^{\prime})$), for every bounding box $b$ labelled (resp. not labelled) with $C_{i}$ and for every pair of bounding boxes $\left<b,b^{\prime}\right>$ connected (resp$\neg\mathsf{partOf}(b,b^{\prime})$. not connected) by the $\mathsf{partOf}$ relation. The partial grounding $\hat{\mathcal{G}}$ is defined on all bounding boxes of all the images in $\mathit{Pics}$ where both the semantic features $class(C_{i},b)$ and the bounding box coordinates are computed by the Fast R-CNN object detector [11]. $\hat{\mathcal{G}}$ is not defined for the predicate symbols in $\mathcal{P}$ and is to be learned. $\mathcal{T}_{\mathrm{expl}}$ contains only assertional information about specific bounding boxes. This is the classical setting of machine learning where classifiers (i.e. the grounding of predicates) are inductively learned from positive examples (such as $\mathsf{partOf}(b,b^{\prime})$) and negative examples ($\neg\mathsf{partOf}(b,b^{\prime})$) of a classification. In this learning setting, mereological constraints such as “cats have no wheels” or “a tail is a part of a cat” are not taken into account. Examples of mereological constraints state, for instance, that the part-of relation is asymmetric ($\forall xy(\mathsf{partOf}(x,y)\rightarrow\neg\mathsf{partOf}(y,x))$), or lists the several parts of an object (e.g. $\forall xy(\mathsf{Cat}(x)\wedge\mathsf{partOf}(x,y)\rightarrow\mathsf{Tail}(y)\vee\mathsf{Muzzle}(y)$)), or even, for simplicity, that every whole object cannot be part of another object (e.g. $\forall xy(\mathsf{Cat}(x)\rightarrow\neg\mathsf{partOf}(x,y)))$ and every part object cannot be divided further into parts (e.g. $\forall xy(\mathsf{Tail}(x)\rightarrow\neg\mathsf{partOf}(y,x)))$. This general knowledge is available from on-line resources, such as WordNet [9], and can be retrieved by inheriting the meronymy relations for every concept correponding to a whole object. A grounded theory that considers also mereological constraints as prior knowledge can be constructed by adding such axioms to $\mathcal{K}_{\mathrm{expl}}$. More formally, we define $\mathcal{T}_{\mathrm{prior}}=\langle\mathcal{K}_{\mathrm{prior}},\hat{\mathcal{G}}\rangle$, where $\mathcal{K}_{\mathrm{prior}}=\mathcal{K}_{\mathrm{expl}}+\mathcal{M}$, and $\mathcal{M}$ is the set of mereological axioms. To check the role of $\mathcal{M}$, we evaluate both theories and then compare results.
在接下来的内容中，我们描述了如何为 SII 构建一个合适的 GT。设 $\mathit{Pics}^{t}\subseteq\mathit{Pics}$ 是一组图像的边界框，这些图像被正确标记为它们所属的类别，并且每对边界框被正确标记为部分关系。在机器学习术语中， $\mathit{Pics}^{t}$ 是一个没有噪声的训练集。在实际语义中，训练集可以用一个理论 $\mathcal{T}_{\mathrm{expl}}=\langle\mathcal{K}_{\mathrm{expl}},\hat{\mathcal{G}}\rangle$ 表示，其中 $\mathcal{K}_{\mathrm{expl}}$ 包含一组封闭文字 $C_{i}(b)$ （或 $\neg C_{i}(b)$ ）和 $\mathsf{partOf}(b,b^{\prime})$ （或 $\neg\mathsf{partOf}(b,b^{\prime})$ ），对于每个被标记（或未被标记）为 $C_{i}$ 的边界框 $b$ ，以及对于每对由 $\mathsf{partOf}$ 关系连接（或未连接）的边界框 $\left<b,b^{\prime}\right>$ 。部分接地 $\hat{\mathcal{G}}$ 定义在 $\mathit{Pics}$ 中所有图像的所有边界框上，其中语义特征 $class(C_{i},b)$ 和边界框坐标都由 Fast R-CNN 目标检测器[11]计算。 $\hat{\mathcal{G}}$ 对 $\mathcal{P}$ 中的谓词符号未定义且需要学习。 $\mathcal{T}_{\mathrm{expl}}$ 仅包含关于特定边界框的断言信息。这是机器学习的经典设置，其中分类器（即。 谓词的基础）是通过正例（例如 $\mathsf{partOf}(b,b^{\prime})$ ）和负例（ $\neg\mathsf{partOf}(b,b^{\prime})$ ）的归类从归纳中学习的。在这种学习环境中，像“猫没有轮子”或“尾巴是猫的一部分”这样的部分整体约束并未被考虑。部分整体约束的示例包括，例如，部分关系是非对称的（ $\forall xy(\mathsf{partOf}(x,y)\rightarrow\neg\mathsf{partOf}(y,x))$ ），或列出了一个物体的几个部分（例如 $\forall xy(\mathsf{Cat}(x)\wedge\mathsf{partOf}(x,y)\rightarrow\mathsf{Tail}(y)\vee\mathsf{Muzzle}(y)$ ），甚至为了简单起见，每个整体物体都不能是另一个物体的一部分（例如 $\forall xy(\mathsf{Cat}(x)\rightarrow\neg\mathsf{partOf}(x,y)))$ ），每个部分物体都不能进一步分解成部分（例如 $\forall xy(\mathsf{Tail}(x)\rightarrow\neg\mathsf{partOf}(y,x)))$ 。这些一般知识可以从在线资源（例如 WordNet [9]）中获得，并且可以通过继承与整体物体对应的每个概念的 meronymy 关系来检索。考虑到部分整体约束的基础理论可以通过将这些公理添加到 $\mathcal{K}_{\mathrm{expl}}$ 中来构建。更正式地，我们定义 $\mathcal{T}_{\mathrm{prior}}=\langle\mathcal{K}_{\mathrm{prior}},\hat{\mathcal{G}}\rangle$ ，其中 $\mathcal{K}_{\mathrm{prior}}=\mathcal{K}_{\mathrm{expl}}+\mathcal{M}$ ，而 $\mathcal{M}$ 是部分整体公理集。为了检查 $\mathcal{M}$ 的作用，我们评估两种理论，然后比较结果。