ST-HOI: A Spatial-Temporal Baseline for
Human-Object Interaction Detection in Videos
ST-HOI：
用于视频中人物交互检测的时空基线

HOI Detection aims to reason over interactions between humans (actors) and target objects. HOI is closely related to visual relationship detection (Lu et al., 2016; Chiou et al., 2021) and scene graph generation (Xu et al., 2017), in which the subject in (subject-predicate-object) are not restricted to a human. HOI in static images has been intensively studied recently (Smith et al., 2013; Chao et al., 2018; Gao et al., 2018; Qi et al., 2018; Li et al., 2019; Wang et al., 2019; Wan et al., 2019; Gupta et al., 2019; Li et al., 2020; Wang et al., 2020; Liao et al., 2020; Ulutan et al., 2020; Zhong et al., 2020). Most of the existing methods can be divided into two categories by the order of human-object pair proposal and interaction classification. The first group (Gao et al., 2018; Chao et al., 2018; Li et al., 2019; Wang et al., 2019; Wan et al., 2019; Gupta et al., 2019; Ulutan et al., 2020; Li et al., 2020) performs human-object pair generation followed by interaction classification, while the second group (Gkioxari et al., 2018; Qi et al., 2018; Wang et al., 2020; Liao et al., 2020) first predicts the most probable interactions performed by a person followed by associating them with the most-likely objects. Our ST-HOI belongs to the first group as we establish a temporal model based on trajectories (continuous object proposals).
HOI 检测旨在对人类（参与者）和目标物体之间的交互进行推理。HOI与视觉关系检测密切相关（Lu et al.， 2016;Chiou et al.， 2021） 和场景图生成 （Xu et al.， 2017），其中主语（主语-谓语-宾语）不限于人类。静态图像中的HOI最近得到了深入研究（Smith等人 ，2013;Chao et al.， 2018;Gao et al.， 2018;Qi et al.， 2018;Li et al.， 2019;Wang et al.， 2019;Wan 等人 ，2019 年;Gupta等人 ，2019;Li et al.， 2020;Wang et al.， 2020;Liao et al.， 2020;Ulutan 等人 ，2020 年;Zhong et al.， 2020）。现有的大多数方法可以按照人-物对、提议和交互分类的顺序分为两类。第一组（Gao et al.， 2018;Chao et al.， 2018;Li et al.， 2019;Wang et al.， 2019;Wan 等人 ，2019 年;Gupta等人 ，2019;Ulutan 等人 ，2020 年;Li 等 .， 2020） 执行人-物体对生成，然后进行交互分类，而第二组 （Gkioxari et al.， 2018;Qi et al.， 2018;Wang et al.， 2020;Liao et al.， 2020） 首先预测一个人执行的最可能的交互，然后将它们与最可能的物体相关联。我们的 ST-HOI 属于第一组，因为我们建立了一个基于轨迹（连续对象建议）的时间模型。

In contrast to the popularity of image-based HOI, there are only a few of studies in VideoHOI (Koppula and Saxena, 2015; Jain et al., 2016; Qi et al., 2018; Sunkesula et al., 2020) and, to the best of our knowledge, all of which conducted experiments on CAD-120 (Koppula et al., 2013) dataset. In CAD-120, the interactions are defined by merely 10 high-level activities (e.g., making cereal or microwaving food) in 120 RGB-D videos. This setting is not favorable to real-life scenarios where machines may be asked to understand more fine-grained actions. Moreover, previous methods (Koppula and Saxena, 2015; Jain et al., 2016; Qi et al., 2018) adopted pre-computed hand-crafted features such as SIFT (Lowe, 2004) which have been outperformed by deep neural networks, and ground truth features including 3D poses and depth information from RGB-D videos which are unlikely to be available in real life scenarios. While (Sunkesula et al., 2020) adopted a ResNet (He et al., 2016) as their backbone, their method is inefficient by requiring $M\times N$ computation for extracting $M$ humans’ and $N$ objects’ features. Different from these existing methods, we evaluate on a larger and more diversified video HOI benchmark dubbed VidHOI, which includes annotations of 50 predicates on thousands of videos. We then propose a spatial-temporal HOI baseline that operates on RGB videos and does not utilize any additional information.
与基于图像的 HOI 的普及相比，VideoHOI 中的研究很少（Koppula 和 Saxena，2015 年;Jain等人 ，2016;Qi et al.， 2018;Sunkesula 等人 ，2020 年），据我们所知，所有这些都在 CAD-120 （Koppula 等人 ，2013 年）数据集上进行了实验。 在 CAD-120 中，交互仅由 120 个 RGB-D 视频中的 10 个高级活动（例如，制作麦片或微波炉加热食物）定义。此设置不利于实际情况，因为在实际生活中，可能会要求计算机理解更细粒度的操作。 此外，以前的方法（Koppula和Saxena，2015;Jain等人 ，2016;Qi et al.， 2018） 采用了预先计算的手工制作特征，例如 SIFT （Lowe， 2004），这些特征已被深度神经网络所超越，以及地面实况特征，包括来自 RGB-D 视频的 3D 姿势和深度信息，这些特征在现实生活中不太可能可用。 虽然（Sunkesula et al.， 2020）采用了ResNet（He et al.， 2016）作为他们的骨干，但他们的方法效率低下，因为需要 $M\times N$ 计算来提取 $M$ 人类和 $N$ 物体的特征。 与这些现有方法不同，我们评估了一个更大、更多样化的视频 HOI 基准，称为 VidHOI，其中包括数千个视频的 50 个谓词的注释。 然后，我们提出了一个时空HOI基线，该基线对RGB视频进行操作，并且不使用任何额外的信息。

STAD aims to localize actors and detect the associated actions (without considering interacting objects). One of the most popular benchmark for STAD is AVA (Gu et al., 2018), where the annotation is done at a sampling frequency of 1 Hz and the performance is measured by framewise mean AP. We followed this annotation and evaluation style when constructing VidHOI, where we converted the original labels into the same format.
STAD旨在定位参与者并检测相关动作（不考虑交互对象）。STAD最受欢迎的基准测试之一是AVA（Gu 等人，2018），其中注释是在1 Hz的采样频率下完成的，性能是通过逐帧平均AP来衡量的。在构建 VidHOI 时，我们遵循了这种注释和评估风格，我们将原始标签转换为相同的格式。

As explained in section 1, a standard approach to STAD (Tran et al., 2015; Carreira and Zisserman, 2017) is extracting spatial-temporal feature maps with a 3D-CNN followed by RoI pooling to crop human features, which are then classified by linear layers. As shown in Figure 2(a), a naive modification that incorporates RoI-pooled human/object features does not work for VideoHOI. In contrast, our ST-HOI tackles VideoHOI by incorporating multiple temporal features including trajectories, correctly-localized visual features and spatial-temporal masking pose features.
如第 1 节所述，STAD 的标准方法（Tran 等人 ，2015 年;Carreira 和 Zisserman，2017 年）正在使用 3D-CNN 提取时空特征图，然后进行 RoI 池化以裁剪人类特征，然后按线性层进行分类。如图 2（a） 所示，包含 RoI 池化人/物体特征的朴素修改不适用于 VideoHOI。相比之下，我们的 ST-HOI 通过结合多种时间特征（包括轨迹、正确定位的视觉特征和时空掩蔽姿势特征）来解决 VideoHOI。

Spatial-Temporal Scene Graph Generation (STSGG) (Ji et al., 2020) aims to generate symbolic graphs representing pairwise visual relationships in video frames. A new benchmark, Action Genome, is also proposed in (Ji et al., 2020) to facilitate researches in STSGG. Ji et al. (Ji et al., 2020) dealt with STSGG by combining off-the-shelf scene graph generation models with long-term feature bank (Wu et al., 2019) on top of a 2D- or 3D-CNN, where they found that the 3D-CNN actually undermines the performance. While observing similar results in VidHOI (Figure 2(a)), we go one step further to find out the underlying reason is that RoI features across frames were erroneously pooled. We correct this by utilizing object trajectories and applying Tube-of-Interest (ToI) pooling on generated trajectories to obtain correctly-localized position information and feature maps throughout video segments.
时空场景图生成 （STSGG） （Ji et al.， 2020） 旨在生成表示视频帧中成对视觉关系的符号图。（Ji et al.， 2020）中还提出了一个新的基准，即Action Genome，以促进STSGG的研究。Ji et al. （Ji et al.， 2020） 通过将现成的场景图生成模型与长期特征库 （Wu et al.， 2019） 结合在 2D 或 3D-CNN 之上来处理 STSGG，他们发现 3D-CNN 实际上破坏了性能。在VidHOI中观察到类似的结果（图2（a））时，我们更进一步，发现根本原因是跨帧的RoI特征被错误地合并。我们通过利用对象轨迹并在生成的轨迹上应用感兴趣管 （ToI） 池化来纠正这一点，以获得在整个视频片段中正确定位的位置信息和特征图。

We follow STAD approaches (Tran et al., 2015; Carreira and Zisserman, 2017; Feichtenhofer et al., 2019) to detect VideoHOI in a keyframe-centric strategy. Denote $V$ as a video which has $T$ keyframes with sampling frequency of 1 Hz as $\{I_{t}\},t=\{1,...,T\}$, and denote $C$ as the number of pre-defined interaction classes. Given $N$ instance trajectories including $M$ human trajectories ($M\leq N$) in a video segment centered at the target frame, for human $m\in\{1,...,M\}$ and object $n\in\{1,...,N\}$ in keyframe $I_{t}$, we aim to detect pairwise human-object interactions $r_{t}=\{0,1\}^{C}$, where each entry $r_{t,c},c\in\{1,...,C\}$ means whether the interaction $c$ exists or not.

Refer to Figure 1(b) for an illustration of our ST-HOI. Our model takes in a video segment (sequence of $T$ frames) centered at $I_{t}$ and utilizes a 3D-CNN as the backbone to extract spatial-temporal feature maps of the whole segment. To rectify the mismatch caused by temporal-RoI pooling, based on $N$ object (including human) trajectories $\{j_{i}\},i=\{1,..,N\},j_{i}\in\mathbb{R}^{T\times 4}$ we generate temporal-aware features including correctly-localized features and spatial-temporal masking pose features. These features together with trajectories are concatenated and classified by linear layers. Note that we aim at a simple but effective temporal-aware baseline to VideoHOI so that we do not utilize tricks in STAD such as non-local block (Wang et al., 2018) or long-term feature bank (Wu et al., 2019), and in image-based HOI like interactiveness (Li et al., 2019), though we note that these may be used to boost the performance.

We have discussed in previous sections on inappropriately pooled RoI features. We propose to tackle this issue by reversing the order of temporal pooling and RoI-pooling. This approach has recently been proposed in (Hou et al., 2017) and named as tube-of-interest pooling (ToIPool). Refer to Figure 3(a) for an illustration. Denote $v\in\mathbb{R}^{d\times T\times H\times W}$ as the output of the penultimate layer of our 3D-CNN backbone, and denote $v_{t}\in\mathbb{R}^{d\times H\times W}$ as the $t$-th feature map along the time axis. Recall that we have $N$ trajectories centered at a keyframe. Following the conventional way, we also exploit visual context when predicting an interaction, which is done by utilizing the union bounding box feature of a human and an object. For example, the sky between human and kite could help to infer the correct interaction fly. Recall that $j_{i}$ represents the trajectory of object $i$, where we further denote $j_{i,t}$ as the 2D bounding box at time $t$. The spatial-temporal instance features $\{\bar{v}_{i}\}$ are then obtained using ToIPool with RoIAlign (He et al., 2017) by

where $\bar{v}_{i}\in\mathbb{R}^{d\times h\times w}$ and $h$ and $w$ means height and width of the pooled feature maps, respectively. $\bar{v}_{i}$ is flattened before concatenating with other features.

Human poses have been widely utilized in image-based HOI methods (Li et al., 2019; Gupta et al., 2019; Wan et al., 2019) to exploit characteristic actor pose to infer some special actions. In addition, some existing works (Wang et al., 2019; Wan et al., 2019) found that spatial information can be used to identify interactions. For instance, for human-ride-horse one can imagine the actor’s skeleton as legs widely open (on horse sides), and the bounding box center of human is usually on top of that of horse. However, none of the existing works consider this mechanism in temporal domain: when riding a horse the human should be moving with horse as a whole. We argue that this temporality is an important property and should be utilized as well.

The spatial-temporal masking pose module is presented at Figure 3(b). Given $M$ human trajectories, we first generate $M$ spatial-temporal pose features with a trained human pose prediction model. On frame $t$, the predicted human pose $h_{i,t}\in\mathbb{R}^{17\times 2},i=\{1,..,M\},t=\{1,..,T\}$ is defined as 17 joint points mapped to the original image. We transform $h_{i,t}$ into a skeleton on a binary mask with $f_{h}:\{h_{i,t}\}\in\mathbb{R}^{17\times 2}\to\{\bar{h}_{i,t}\}\in\mathbb{R}^{1\times H\times W}$, by connecting the joints using lines, where each line has a distinct value $x\in[0,1]$. This helps the model to recognize and differentiate different poses.

For each of $M\times(N-1)$ valid human-object pairs on frame $t$, we also generate two spatial masks $s_{i,t}\in\mathbb{R}^{2\times H\times W},i=\{1,...,M\times(N-1)\}$ corresponding to human and object respectively, where the values inside of each bounding box are ones and outsides are zeroed-out. These masks enable our model to predict HOI with reference to important spatial information.

For each pair, we concatenate the skeleton mask $\bar{h}_{i,t}$ and spatial masks $s_{i,t}$ along the first dimension to get the initial spatial masking pose feature $p_{i,t}\in\mathbb{R}^{3\times H\times W}$:

We then down-sample $\{p_{i,t}\}$, feed into two 3D convolutional layers with spatial and temporal pooling, and flatten to obtain the final spatial-temporal masking pose feature $\{\bar{p}_{i,t}\}$.

We fuse the aforementioned features, including correctly-localized visual features $\bar{v}$, spatial-temporal masking pose features $p$, and instance trajectories $j$ by concatenating them along the last axis

where we slightly abuse the notation to denote the subscriptions $s$ as the subject, $o$ as the object and $u$ as their union region. $v_{\text{so}}$ is then fed into two linear layers with the final output size being the number of interaction classes in the dataset. Since VideoHOI is essentially a multi-label learning task, we train the model with per-class binary cross entropy loss.

During inference, we follow the heuristics in image-based HOI (Chao et al., 2018) to sort all the possible pairs by their softmax scores and evaluate on only top 100 predictions.

While we have discussed in section 2.1 about the problem of lacking a suitable VideoHOI dataset by analyzing CAD-120 (Koppula et al., 2013), we further explain why Action Genome (Ji et al., 2020) is also not a feasible choice here. First, the authors acknowledged that the dataset is still incomplete and contains incorrect labels (Ji, 2020). Second, Action Genome is produced by annotating Charades (Sigurdsson et al., 2016), which is originally designed for activity classification where each clip contains only one ”actor” performing predefined tasks; should any other people show up, there are neither any bounding box nor interaction label about them. Finally, the videos are purposedly-generated by volunteers, which are rather unnatural. In contrast, VidHOI are based on VidOR (Shang et al., 2019) which is densely annotated with all humans and predefined objects showing up in each frame. VidOR is also more challenging as the videos are non-volunteering user-generated and thus jittery at times. A comparison of VidHOI and the existing STAD and HOI datasets is presented in Table 1.

VidOR is originally collected for video visual relationship detection where the evaluation is trajectory-based. The volumetric Interaction Over Union (vIOU) between a trajectory and a ground truth needs to be over 0.5 before considering its relationship prediction; however, how to obtain accurate trajectories with correct start- and end-timestamp remains challenging (Sun et al., 2019; Shang et al., 2017). We notice that some image-based HOI datasets (e.g., HICO-DET (Chao et al., 2018) and V-COCO (Gupta and Malik, 2015)) as well as STAD datasets (e.g., AVA (Gu et al., 2018)) are using a keyframe-centered evaluation strategy, which bypasses the aforementioned issue. We thus adopt the same and follow AVA to sample keyframes at a 1 FPS frequency, where the annotations on the keyframe at timestamp $t$ are assumed to be fixed for $t\pm 0.5$s. In detail, we first filter out those keyframes without presenting at least one valid human-object pair, followed by transforming the labels from video clip-based to keyframe-based to align with common HOI metrics (i.e., frame mAP). We follow the original VidOR split in (Shang et al., 2019) to divide VidHOI into a training set comprising 193,911 keyframes in 6,366 videos and a validation set²²2The VidOR testing set is not available publicly. with 22,808 keyframes in 756 videos. As shown in Figure 4, there are 50 relation classes including actions (e.g., push, pull, lift, etc.) and spatial relations (e.g., next to, behind, etc.). While half (25) of the predicate classes are temporal-related, they account for merely $\sim$5% of the dataset.

Following the evaluation metric in HICO-DET, we adopt mean Average Precision (mAP), where a true positive HOI needs to meet three below criteria: (a) both the predicted human and object bounding boxes have to overlap with the ground truth boxes with IOU over 0.5, (b) the predicted target category need to be matched and (c) the predicted interaction is correct. Over 50 predicates, we follow HICO-DET to define HOI categories as 557 triplets on which we compute mean AP. By defining HOI categories with triplets we can bypass the polysemy problem (Zhong et al., 2020), i.e., the same predicate word can represent very different meaning when pairing with distinct objects, e.g., person-fly-kite and person-fly-airplane. We report the mean AP over three categories: (a) Full: all 557 categories are evaluated, (b) Rare: 315 categories with less than 25 instances in the dataset, and (c) Non-rare: 242 categories with more than or equal to 25 instances in the dataset. We also examine the models in two evaluation modes: Oracle models are trained and tested with ground truth trajectories, while models in Detection mode are tested with predicted trajectories.

We adopt Resnet-50 (He et al., 2016) as our 2D backbone for the preliminary experiments, and utilize Resnet-50-based SlowFast (Feichtenhofer et al., 2019) as our 3D backbone for all the other experiments. SlowFast contains the Slow and Fast pathways, which correspond to the texture details and the temporal information, respectively, by sampling video frames in different frequencies. For a 64-frame segment centered at the keyframe, $T=32$ frames are alternately sampled to feed into the Slow pathway; only $T/\alpha$ frames are fed into the Fast pathway, where $\alpha=8$ in our experiments. We use FastPose (Fang et al., 2017) to predict human poses and adopt the predicted trajectories generated by a cascaded model of video object detection, temporal NMS and tracking algorithm (Sun et al., 2019). Like object detection is to 2D HOI detection, trajectory generation is an essential module but not a main focus of this work. If a bounding box is not available in neighboring frames (i.e., the trajectory is shorter than $T$ or not continuous throughout the segment), we fill it with the whole-image as a box. We train all models from scratch for 20 epochs with the initial learning rate $1\times 10^{-2}$, where we use step decay learning rate to reduce the learning rate by $10\times$ at the $10^{\text{th}}$ and $15^{\text{th}}$ epoch. We optimize our models using synchronized SGD with momentum of 0.9 and weight decay of $10^{-7}$. We train each 3D video model with eight NVIDIA Tesla V100 GPUs with batch size being 128 (i.e., 16 examples per GPU), except for the full model where we set batch size as 112 due to the memory restriction. We train the 2D model with a single V100 with batch size being 128.

During training, following the strategy in SlowFast we randomly scale the shorter side of the video to a value in $[256,320]$ pixels, followed by random horizontal flipping and random cropping into $224\times 224$ pixels. During inference, we only resize the shorter side of the video segment to 224 pixels.

Since we aim to deal with a) the lack of temporal-aware features in 2D HOI methods, b) the feature inconsistency issue in common 3D HOI methods and c) the lack of a VideoHOI benchmark, we mainly compare with the 2D model (Wan et al., 2019) and its naive 3D variant on VidHOI to understand if our ST-HOI addresses these issues effectively.

The performance comparison between our full ST-HOI model (Ours-T+V+P) and baselines (2D model, 3D model) are presented in Table LABEL:tab:results, in which we also present ablation studies on our different features (modules) inlcuding trajectory features (T), correctly-localized visual features (V) and spatial-temporal masking pose features (P). Table LABEL:tab:results shows that 3D model only has a marginal improvement compared to 2D model (overall $\sim$2%) under all settings in both evaluation modes. In contrast, adding trajectory features (Ours-T) leads to a much larger 23% improvement in Oracle mode or 15% in Detection mode, showing the importance of correct spatial-temporal information. We also find that by adding additional temporal-aware features (i.e., V and P) increasingly higher mAPs are attained, and our full model (Ours-T+V+P) reports the best mAPs in Oracle mode, achieving the highest $\sim$25% relative improvement. We notice that the performance of Ours-T+V is close to that of Ours-T under Oracle setting, which is possibly because the ground truth trajectories (T) have provided enough “correctly-localized” information so that the correct features do not help much. We also note that the performance of Ours-T+P is slightly higher than that of Ours-T+V+P under Detection mode, which is assumably due to the same, aforementioned reason and the inferior performance resulting from the predicted trajectories. The overall performance gap between Detection and Oracle models is significant, indicating the room for improvement in trajectory generation. Another interesting observation is that Full mAPs are very close to Rare mAPs, especially under Oracle mode, showing that the long-tail effect over HOIs is strong (but common and natural).

To understand the effect of temporal features on individual predicates, we compare with predicate-wise AP (pAP) shown in Figure 5. We observe that, again, under most of circumstances naively replacing 2D backbones with 3D ones does not help video HOI detection. Both temporal predicates (e.g., towards, away, pull) and spatial (e.g., next_to, behind, beneath) predicates benefit from the additional temporal-aware features in ST-HOI. These findings verify our main idea about the essential use of trajectories and trajectory-based features. In addition, each additional features do not seem to contribute equally for different predicates. For instance, we see that while Ours-T+V+P performs the best on some predicates (e.g., behind and beneath), our sub-models achieve the highest mAP on other predicates (e.g., watch and ride). This is assumedly because predicate-wise performance is heavily subject to the number of examples, where major predicates have 10-10000 times more examples than minor ones (as shown in Figure 4).

Since the majority of HOI examples are spatial-related ($\sim$95%, as shown in Figure 4), the results above might not be suitable for demonstrating the temporal modeling ability of our proposed model. We thus focus on the performance on only temporal-related predicates in Figure 6, which demonstrates that Ours-T+V+P greatly outperforms the baselines regarding the top frequent temporal predicates. Table LABEL:tab:temporal_spatial_results presents the triplet mAPs of spatial- or temporal-only predicates, showing Ours-T significantly improves the 2D model on temporal-only mAP by relative +73.9%, in sharp contrast to -7.1% of the 3D model in Oracle mode. Similar to our observation with Table LABEL:tab:results, Ours-T performs on par with Ours-T+V+P for temporal-only predicates; however, it falls short of spatial-only predicates, showing that spatial/pose information is still essential for detecting spatial predicates. Overall, these results demonstrate the outstanding spatial-temporal modeling ability of our approach.

We also compare the performance with respect to some HOI triplets in Figure 7. Similar to the results on predicate-wise mAP, we also observe the large gap between naive 2D/3D models and our models with the temporal features. ST-HOI variants are more accurate in predicting especially temporal-aware HOIs (hug/lean_on-person and push/pull-baby_walker). We also see in some examples that Ours-T+V+P does not perform the best among all the variants, e.g., lean_on-person), which is similar to the phenomenon we observed in Figure 5.

To understand the effectiveness of our proposed method, we visualize two video HOI examples of VidHOI predicted by the 2D model (Wan et al., 2019) and Ours-T+V+P (both in Oracle mode) in Figure 8. Each (upper and lower) example is a 5-second video segment (i.e., five keyframes) with a HOI prediction table where each entry means either True Positive (TP), False Positive (FP), False Negative (FN) or True Negative (TN) for both models. The upper example shows that, compared to the 2D model, Ours-T+V+P makes more accurate HOI detection by successfully predicting hold_hand_of at $T_{4}$ and $T_{5}$. Moreover, Ours-T+V+P is able to predict interactions that requires temporal information, such as lift at $T_{1}$ in the lower example. However, we can see that there is still room for improvement for Ours-T+V+P in the same example, where lift is not detected in the following $T_{2}$ to $T_{4}$ frames. Overall, our model produces less false positives throughout the dataset, which in turn contributes to its higher mAP and pAP.