PROTOtypical Logic Tensor Networks (PROTO-LTN) for Zero Shot Learning
零样本学习的原型逻辑张量网络（PROTO-LTN）

Research on how to combine connectionist and symbolic approaches has flourished in the past few years [5, 12], with several applications in semantic image interpretation and visual query answering [5, 4, 13, 3, 14, 15, 16]. Among the plethora of compositional patterns that have been proposed [17, 12], the present work follows two main principles: knowledge representation (in the form of first order logic) is embedded into a neural network, which in turn allows to constrain the search space by leveraging explicit (and human-interpretable) domain knowledge as a symbolic prior. This latter property is extremely useful in ZSL, in which some external source of information is exploited to offer an abstract description of the classes in lieu of providing training examples. On the other hand, compared to approaches based on Inductive Logic Programming (such as [14]), in which perception and reasoning are performed by separate modules, LTNs provide tighter integration between the two subsystems.
过去几年来，关于如何结合连接主义和符号方法的研究蓬勃发展[5, 12]，在语义图像解释和视觉查询回答等多个应用中取得了一些成果[5, 4, 13, 3, 14, 15, 16]。在众多已被提出的组合模式中[17, 12]，本研究遵循两个主要原则：将知识表示（以一阶逻辑形式）嵌入到神经网络中，从而通过利用显式（且人类可解释的）领域知识作为符号先验来约束搜索空间。后一属性在零样本学习中非常有用，其中利用某些外部信息源来提供类别的抽象描述，而不是提供训练样本。另一方面，与基于归纳逻辑编程（如[14]）的方法相比，其中感知和推理由独立模块执行，逻辑张量网络提供了两个子系统之间更紧密的集成。

LTNs have proven effective in higher-level image interpretation tasks, such as object detection and scene graph construction [13, 5]. Donadello et al. applied them for scene relationship detection in a zero shot setting, showing how prior knowledge can compensate for the lack of supervision [3].
LTNs 已被证明在更高级别的图像解释任务中非常有效，比如目标检测和场景图构建[13, 5]。Donadello 等人将其应用于零样本设置中的场景关系检测，展示了先前知识如何弥补监督缺失[3]。

In the LTN framework, the term grounding denotes the interpretation of a First Order Language into a subset of the $\mathbb{R}^{n}$ domain [5]. It defines a collection of terms (objects) and formulas described in a Knowledge base $\mathcal{K}$. For instance, to express the friendship between two terms defined as Alice and Bob, we can use the predicate friend_of:
在 LTN 框架中，接地术语表示将一阶语言解释为 $\mathbb{R}^{n}$ 域的子集[5]。它定义了在知识库 $\mathcal{K}$ 中描述的一组术语（对象）和公式。例如，要表达定义为 Alice 和 Bob 的两个术语之间的友谊，我们可以使用谓词 friend_of：

At the same time, we can specify formulas defining general properties, such as the symmetric nature of the friendship relationship within a specific domain:
与此同时，我们可以指定定义一般属性的公式，比如在特定领域内友谊关系的对称性质：

Adopting Real Logic, both formulas and terms are grounded (interpreted) into a scalar value in the [0,1] interval. Specifying the grounding function $\mathcal{G}$, which maps terms and formulas into such real-valued features, generates a complete definition of a theory. Given a set of terms, aggregate formulas can be defined by approximating unary, binary or quantifiers connectives in fuzzy logic using suitable differential operators.
采用真实逻辑，无论是公式还是术语都被基于[0,1]区间内的标量值进行了基础化（解释）。指定将术语和公式映射为这种实值特征的基础化函数 $\mathcal{G}$ ，生成了一个理论的完整定义。给定一组术语，可以通过使用适当的微分算子来近似模糊逻辑中的一元、二元或量词连接符，定义聚合公式。

In semantic image interpretation tasks, terms (objects) are typically grounded by features computed by a pre-trained convolutional neural network; it is also possible to jointly train the convolutional backbone and the LTNs in an end-to-end fashion [4]. Predicates symbols $p\in\mathcal{P}$ are grounded by a function $\mathcal{G}\left(D(p)\right)\rightarrow[0,1]$. A typical predicate in semantic image interpretation is the isOfClass one, which represents the probability that a given object belongs to class $c$.
在语义图像解释任务中，术语（对象）通常由预先训练的卷积神经网络计算的特征进行基础化；也可以以端到端的方式联合训练卷积主干和 LTNs [4]。谓词符号 $p\in\mathcal{P}$ 由函数 $\mathcal{G}\left(D(p)\right)\rightarrow[0,1]$ 进行基础化。在语义图像解释中的典型谓词是 isOfClass，它表示给定对象属于类 $c$ 的概率。

In conventional LTNs [5, 13, 4], predicates are typically defined as the generalization of the neural tensor network:
在传统的 LTNs [5, 13, 4] 中，谓词通常被定义为神经张量网络的泛化：

where $\sigma$ is the sigmoid function, $W[1:k]\in\mathbb{R}^{k\times mn\times mn}$, $V_{p}\in\mathbb{R}^{k\times mn}$, $u_{p}\in\mathbb{R}^{k}$, and $b_{p}\in\mathbb{R}$ are learnable tensors of parameters. For multi-class problems, the sigmoid function could be substituted by a softmax layer to enforce mutual exclusivity [5].
其中 $\sigma$ 是 Sigmoid 函数， $W[1:k]\in\mathbb{R}^{k\times mn\times mn}$ ， $V_{p}\in\mathbb{R}^{k\times mn}$ ， $u_{p}\in\mathbb{R}^{k}$ 和 $b_{p}\in\mathbb{R}$ 是可学习的参数张量。对于多类问题，Sigmoid 函数可以被 Softmax 层替换以强制互斥性。

This grounding requires to add an additional predicate for each class (e.g., isDog, isPerson, etc.), which is embedded into a tensor network with separate weights. Additionally, since class symbols are not grounded, predicates can only be defined for object instances, which rapidly leads to very large knowledge bases when background logical axioms need to be imposed. On the contrary, our proposed grounding does not require additional model parameters, or in any case limits them to a small set which is shared among all isOfClass predicates. Furthermore, it encodes abstract classes as parametric objects that live in the same embedding space as their instances, and can be used to establish relationships with other objects (e.g., macro-category relationships). This formulation thus supports more efficient and compact representations.
这种基础要求为每个类别添加一个额外的谓词（例如，isDog，isPerson 等），该谓词嵌入到一个具有独立权重的张量网络中。此外，由于类别符号没有基础，谓词只能针对对象实例进行定义，当需要施加背景逻辑公理时，这很快会导致非常庞大的知识库。相反，我们提出的基础不需要额外的模型参数，或者在任何情况下将其限制为一小组共享在所有 isOfClass 谓词之间的参数。此外，它将抽象类别编码为生存在与其实例相同的嵌入空间中的参数化对象，并可用于与其他对象建立关系（例如，宏类别关系）。因此，这种表述支持更高效和紧凑的表示。

The best satisfability problem, which is the optimization problem underlying LTNs, consists in determining the values of $\Theta^{*}$ that maximize the truth values of the conjunction of all formulas $\phi\in\mathcal{K}$:
最佳可满足性问题是逻辑张量网络（LTNs）的基础优化问题，其目标是确定最大化所有公式的合取真值的 $\Theta^{*}$ 的值： $\phi\in\mathcal{K}$

where $\lambda||\Theta||_{2}^{2}$ is a convenient regularization term.
其中 $\lambda||\Theta||_{2}^{2}$ 是一个方便的正则化项。

In zero-shot learning, a learner must be able to recognize objects from test classes, not seen during training, by leveraging some sort of description, most commonly a vector of semantic attributes [18]. In this paper, we target the Generalized zero-shot learning (GZSL) scenario, in which both seen and unseen classes appear at test time [18]. State-of-the-art techniques for ZSL classification typically fall within two categories [18, 8]: embedding-based and generative-based.
在零样本学习中，学习者必须能够通过利用某种描述（最常见的是语义属性向量）来识别训练过程中未见过的测试类别中的对象。本文针对广义零样本学习（GZSL）场景，其中在测试时出现了已见和未见类别。目前用于零样本学习分类的先进技术通常分为两类：基于嵌入的和基于生成的。

Embedding-based models [8, 19, 20, 21] compare semantic characteristics (e.g., attributes) and visual characteristics (usually taken from a pre-trained convolutional neural network) by (learning a) mapping to a common embedding space. Mapping the semantic space to the more compact visual feature space, rather than the opposite, alleviates the so-called hubness problem and facilitates separation between classes [8]. Standard embedding-based models are completely agnostic to any information about the test set: neither examples (even unlabelled), nor class attributes are assumed to be available at training time. Although based on a NeSy formulation, the proposed PROTO-LTN approach can be regarded as an embedding-based technique, as semantic concepts and visual features are mapped onto a common embedding space.
基于嵌入的模型[8, 19, 20, 21]通过将语义特征（例如属性）和视觉特征（通常来自预先训练的卷积神经网络）映射到一个共同的嵌入空间中进行比较。将语义空间映射到更紧凑的视觉特征空间，而不是相反，有助于缓解所谓的中心性问题，并促进类别之间的分离[8]。标准的基于嵌入的模型对测试集的任何信息都是完全不可知的：在训练时既不假设有示例（甚至未标记的），也不假设有类别属性。尽管基于 NeSy 公式，所提出的 PROTO-LTN 方法可以被视为一种基于嵌入的技术，因为语义概念和视觉特征被映射到一个共同的嵌入空间中。

Embedding-based models tend to be naturally biased towards seen classes. To alleviate this problem, generative models were proposed with the purpose of learning a conditioned probability distribution for each class, and thus generate artificial examples of unseen classes [22, 23, 24]. A conventional classifier is trained by utilizing both the true and the generated examples. Although impressive results, especially in a GZSL context, can be achieved by taking advantage of this machinery, reduced flexibility with respect to embedding methods is entailed, as unseen classes need to be defined, so that a number of corresponding examples can be artificially synthesized. PROTO-LTNs are thus best compared with other embedding-based models, although nothing prevents them from being trained on, or combined with, generative methods.
基于嵌入的模型往往在已知类别上具有自然偏见。为了缓解这一问题，提出了生成模型，目的是为每个类别学习一个条件概率分布，从而生成未知类别的人工示例[22, 23, 24]。传统分类器通过利用真实和生成的示例进行训练。尽管通过利用这种机制，特别是在广义零样本学习的情况下可以取得令人印象深刻的结果，但与嵌入方法相比，存在着较少的灵活性，因为需要定义未知类别，以便人工合成相应数量的示例。因此，PROTO-LTNs 最好与其他基于嵌入的模型进行比较，尽管它们也可以在生成方法上进行训练或结合。

A $N$-way-$K$-shot FSL scenario is supposed, in which a classifier is asked to discriminate the right class among $N$ choices, while having the chance to observe $K$ examples per class [25, 26, 27]. More specifically, the labelled examples are referred to as the support examples, whereas the unlabeled ones as the query examples.
一种 $N$ -way- $K$ -shot FSL 场景被假设，其中分类器被要求在 $N$ 个选择中区分正确的类别，同时有机会观察每个类别的 $K$ 个示例[25, 26, 27]。更具体地说，标记的示例被称为支持示例，而未标记的示例被称为查询示例。

The underlying assumption that it exists an embedding space in which elements of different classes are well-scattered, and that it can be mathematically translated into an embedding function $f_{\theta}$ whose parameter $\theta$ must be inferred, acting as a mapping
存在这样一个嵌入空间的基本假设，其中不同类别的元素被很好地分散，并且可以在数学上转化为一个嵌入函数 $f_{\theta}$ ，其参数 $\theta$ 必须被推断出来，充当映射

In Eq. 3, $D$ and $M$ are, respectively, the dimensions of the input space and of the embedding space. Thus, for an example $x$, $f_{\theta}(x)$ is the corresponding embedding.
在方程 3 中， $D$ 和 $M$ 分别是输入空间和嵌入空间的维度。因此，对于一个示例 $x$ ， $f_{\theta}(x)$ 是相应的嵌入。

In FSL, a prototype for class $n$ is obtained as the mean embedding of the $K$ support examples of class $n$ at train time:
在 FSL 中，类 $n$ 的原型是在训练时由类 $n$ 的 $K$ 个支持示例的平均嵌入获得的：

Class prototypes thus need to live in the embedding space, as they embody average features shared by elements of the class they represent. At training time, $\theta$ is optimized so that the distance between each prototype and the elements of its class is minimized, while the distance between different prototypes is maximized. Finally, classification at testing time is performed by assigning each query sample to its nearest prototype.
类原型因此需要存在于嵌入空间中，因为它们体现了它们所代表的类的元素共享的平均特征。在训练时， $\theta$ 被优化，以使每个原型与其类的元素之间的距离最小化，同时不同原型之间的距离最大化。最后，在测试时，通过将每个查询样本分配给其最近的原型来进行分类。

At testing time, a support set is at disposal of $N_{S}$ labeled examples $S=\{(x^{S}_{1},y^{S}_{1}),...,(x^{S}_{N_{S}},y^{S}_{N_{S}})\}$, where each $x^{S}_{i}\in\mathbb{R}^{D}$ is the feature vector of an example, and $y^{S}_{i}\in C\subset\mathbb{N}$ is the corresponding label. Assuming a $N$-way-$K$-shot scenario, exactly $K$ support examples are available for each of the $N$ classes. A query set $Q=\{x^{Q}_{1},...,x^{Q}_{N_{Q}}\}$ of $N_{Q}$ unlabeled examples is thus supplied, and the task is to correctly assort the examples into their classes. The elements from the query set $Q$ belong to the same domain as those from the support set $S$.
在测试时，支持集中有 $N_{S}$ 个标记示例 $S=\{(x^{S}_{1},y^{S}_{1}),...,(x^{S}_{N_{S}},y^{S}_{N_{S}})\}$ ，其中每个 $x^{S}_{i}\in\mathbb{R}^{D}$ 是一个示例的特征向量， $y^{S}_{i}\in C\subset\mathbb{N}$ 是相应的标签。假设一个 $N$ -路- $K$ -拍摄场景，每个 $N$ 类别都有 $K$ 个支持示例可用。因此，提供了一个包含 $N_{Q}$ 个未标记示例的查询集 $Q=\{x^{Q}_{1},...,x^{Q}_{N_{Q}}\}$ ，任务是正确地将示例分类。查询集 $Q$ 中的元素属于与支持集 $S$ 中的元素相同的域。

At training time, it could be impossible to know which classes will the testing scenario yield. In other words, a support set $S$ is not accessible in advance. To cope with that, a training set $T=\{(x^{T}_{1},y^{T}_{1}),...,(x^{T}_{N_{T}},y^{T}_{N_{T}})\}$ is chosen that reflects the best prior information possessed about the testing scenario, with labels $y_{i}^{T}\in C_{T}\subset\mathbb{N}$ and $|C_{T}|=N_{T}$ classes which can coincide or outnumber them ($N_{T}\geq N$). In other words, it is possible that $C\cap C_{T}\neq\emptyset$, but it cannot be said in advance. Then, fake support and query sets $\tilde{S}\subset T$ and $\tilde{Q}\subset T$ are extracted to mimic the testing scenario and instruct the model to learn accordingly.
在训练时，可能无法知道测试场景将产生哪些类别。换句话说，支持集 $S$ 事先不可访问。为了应对这种情况，选择一个训练集 $T=\{(x^{T}_{1},y^{T}_{1}),...,(x^{T}_{N_{T}},y^{T}_{N_{T}})\}$ ，反映对测试场景拥有的最佳先验信息，具有标签 $y_{i}^{T}\in C_{T}\subset\mathbb{N}$ 和 $|C_{T}|=N_{T}$ 类别，这些类别可以与之重叠或超过它们（ $N_{T}\geq N$ ）。换句话说，可能会 $C\cap C_{T}\neq\emptyset$ ，但无法事先说出。然后，提取虚假的支持和查询集 $\tilde{S}\subset T$ 和 $\tilde{Q}\subset T$ ，以模仿测试场景并指导模型相应地学习。

In ZSL, one does not dispose of labelled examples for all classes. Instead, it is assumed that $N$ abstract vectors denoted as $\{a^{(1)},a^{(2)},...,a^{(N)}\}$, with $a^{(n)}\in\mathbb{R}^{A}$, encode the characteristics of all $N$ classes.
在 ZSL 中，一个不会为所有类别丢弃标记的示例。相反，假定有 $N$ 个抽象向量表示为 $\{a^{(1)},a^{(2)},...,a^{(N)}\}$ ，具有 $a^{(n)}\in\mathbb{R}^{A}$ ，编码了所有 $N$ 类别的特征。

As in FSL, at training time one takes advantage of a set $T=\{(x^{T}_{1},y^{T}_{1}),...,(x^{T}_{N_{T}},y^{T}_{N_{T}})\}$ of labelled examples from classes $y^{T}_{i}\in C_{T}\subset\mathbb{N}$, where it is preferably $|C_{T}|=N_{T}\geq N=|C|$. The training cycle remains unchanged in the ZSL case, but class prototypes are defined differently:
与 FSL 一样，在训练时，人们利用来自类 $y^{T}_{i}\in C_{T}\subset\mathbb{N}$ 的一组标记示例 $T=\{(x^{T}_{1},y^{T}_{1}),...,(x^{T}_{N_{T}},y^{T}_{N_{T}})\}$ ，最好是 $|C_{T}|=N_{T}\geq N=|C|$ 。在 ZSL 情况下，训练周期保持不变，但类原型的定义不同：

• •

  the embedding for a query example $x^{Q}$ is still obtained as $f_{\theta}(x^{Q})$, where $f_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{M}$;

  
  • 查询示例 $x^{Q}$ 的嵌入仍然是通过 $f_{\theta}(x^{Q})$ 获得的，其中 $f_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{M}$ ；
• •

  the prototype for class $n\in C$ is extracted as $p_{n}=g_{\theta}(a^{(n)})$ via a separate embedding function $g_{\theta}:\mathbb{R}^{A}\to\mathbb{R}^{M}$, which maps the semantic attribute space to the common embedding space.

  
  • 类别 $n\in C$ 的原型通过单独的嵌入函数 $g_{\theta}:\mathbb{R}^{A}\to\mathbb{R}^{M}$ 提取为 $p_{n}=g_{\theta}(a^{(n)})$ ，该函数将语义属性空间映射到共同的嵌入空间。

The overall architecture of PROTO-LTN, when tailored to the ZSL scenario, is illustrated in Fig. 1. The input image embeddings are extracted from a convolutional neural network, while attribute vectors are mapped into the embedding domain through an embedding function. In this section, details about the definition of the grounding of the constant, variables, functions and predicates are given. Then, the Knowledge Base $\mathcal{K}$ which encodes our learning problem is defined.
PROTO-LTN 的整体架构，当定制为零样本学习场景时，如图 1 所示。输入图像嵌入是从卷积神经网络中提取的，而属性向量通过嵌入函数映射到嵌入域中。在本节中，给出了关于常量、变量、函数和谓词接地定义的细节。然后，定义了编码我们学习问题的知识库 $\mathcal{K}$ 。