A survey of human-computer interaction (HCI) & natural habits-based behavioural biometric modalities for user recognition schemes
针对用户识别方案的人机交互 (HCI) 和基于自然习惯的行为生物识别模式的调查

1.1. Objectives and survey strategy
1.1.目标和调查策略

The objective of this article is to survey HCI and natural habits-based biometrics that can be utilized by researchers and engineers to design uni-modal or multi-modal user recognition schemes (leveraging concepts such as implicit, continuous, or risk-based  [9]) for security-sensitive applications, thus, safeguarding IoT ecosystems. Fig. 3 illustrates the timeline and Table 1 lists previous surveys related to the behavioural biometric modalities covered in this article.
本文的目的是调查人机交互和基于自然习惯的生物识别技术，研究人员和工程师可以利用这些生物识别技术来设计单模式或多模式用户识别方案（利用隐式、连续或基于风险等概念[9] ]）适用于安全敏感的应用程序，从而保护物联网生态系统。图 3 说明了时间线，表 1 列出了之前与本文涵盖的行为生物识别模式相关的调查。

1. Download : Download high-res image (965KB)
   下载：下载高分辨率图像 (965KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 3. Timeline showing the important events related to behavioural biometric modalities covered in this article.
图 3. 时间线显示了与本文涵盖的行为生物识别模式相关的重要事件。

Table 1. Earlier behavioural biometrics surveys.
表 1. 早期的行为生物识别调查。

Ref	Year 年	Contributions 贡献
Yampolskiy & Govindaraju [24] 扬波尔斯基和戈文达拉茹 [24]	2008	This survey presented a classification of behavioural biometrics based on skills, style, preference, knowledge, motor skills, or strategy applied by humans. 这项调查根据人类应用的技能、风格、偏好、知识、运动技能或策略对行为生物识别进行了分类。
Meng et al. [25] 孟等人。 [25]	2015	This survey covered the development of biometric user authentication techniques on mobile phones. And, presented a study of voice, signature, gait, behavior profiling, keystroke and touch dynamics behavioural biometrics. 该调查涵盖了移动电话上生物识别用户身份验证技术的发展。并且，提出了对语音、签名、步态、行为分析、击键和触摸动态行为生物识别的研究。
Alzubaidi & Kalita [26] 阿尔祖拜迪和卡利塔 [26]	2016	This survey investigated authentication of smartphone users based on handwaving, gait, touchscreen, keystroke, voice, signature and general profiling behavioural biometrics. 这项调查调查了基于挥手、步态、触摸屏、击键、语音、签名和一般分析行为生物识别技术的智能手机用户身份验证。
Oak [27] 橡木 [27]	2018	This survey analyzed persons’ behavior, such as keystroke dynamics, mouse dynamics, haptics, gait, and log files, for their designing persistent security solutions. 这项调查分析了人们的行为，例如击键动力学、鼠标动力学、触觉、步态和日志文件，以便他们设计持久的安全解决方案。
Dang et al. [28] 党等人。 [28]	2020	This survey focused on Human activity recognition (HAR) for designing context-aware applications for emerging domains like IoT and healthcare by analyzing sensor- and vision-based behavioural patterns. 这项调查的重点是人类活动识别 (HAR)，通过分析基于传感器和视觉的行为模式，为物联网和医疗保健等新兴领域设计上下文感知应用程序。
Stylios et al. [29] 斯蒂利奥斯等人。 [29]	2020	This survey presented the classification of behavioural biometrics technologies. It reviewed behavioural traits like gait, touch gestures, keystroke dynamics, hand-waving, behavioural profile, power consumption, for continuous authentication for mobile devices. 这项调查提出了行为生物识别技术的分类。它审查了步态、触摸手势、击键动态、挥手、行为概况、功耗等行为特征，以实现移动设备的持续身份验证。

In this survey, we first elucidate attributes and features of behavioural biometric modalities that can be acquired from smart devices equipped with motion sensors, touch screens, and microphones or by external IoT sensors or nodes in an unobtrusive manner. We discuss the methodologies, classifiers, datasets, and performance results of recent user recognition schemes that employ these behavioural biometrics modalities. We then present security, privacy, and usability attributes with regard to the CIA properties in human-to-things recognition schemes. Ultimately, the challenges, limitations, prospects, and opportunities associated with behavioural biometric-based user recognition schemes are presented.
在本次调查中，我们首先阐明行为生物识别模式的属性和特征，这些属性和特征可以从配备运动传感器、触摸屏和麦克风的智能设备或通过外部物联网传感器或节点以不显眼的方式获取。我们讨论了采用这些行为生物识别模式的最新用户识别方案的方法、分类器、数据集和性能结果。然后，我们介绍人对物识别方案中 CIA 属性的安全性、隐私性和可用性属性。最后，提出了与基于行为生物识别的用户识别方案相关的挑战、局限性、前景和机遇。

1.2. Article structure 1.2.文章结构

The article is structured as follows: Section 2 discusses behavioural biometrics, sensors, human-to-things recognition mechanisms and performance metrics. Section 2.1 elicits attributes and features of touch-stroke, swipe, touch signature, hand-movements, voice, gait, and footstep modalities that can be exploited for designing user recognition schemes. Section 4 presents the state-of-the-arts of user recognition schemes based on modalities discussed in Section 2.1. Section 5 presents a discussion on security, privacy, and usability of behavioural biometric-based user recognition schemes. Section 6 discusses the open challenges and limitations that deserve attention together with prospects and opportunities for evolving and designing behavioural biometric-based human-to-things recognition schemes. Section 7 concludes the article.
本文的结构如下：第 2 部分讨论行为生物识别、传感器、人对物识别机制和性能指标。第 2.1 节引出了触摸、滑动、触摸签名、手部动作、语音、步态和脚步模式的属性和特征，可用于设计用户识别方案。第 4 节介绍了基于第 2.1 节中讨论的模式的最先进的用户识别方案。第 5 节讨论了基于行为生物识别的用户识别方案的安全性、隐私性和可用性。第 6 节讨论了值得关注的开放挑战和限制，以及发展和设计基于行为生物识别的人对物识别方案的前景和机遇。第 7 节总结了本文。

2.1. Behavioural biometrics
2.1.行为生物识别

Behavioural biometrics involve human behavioural characteristics or activity patterns that are measurable and uniquely identifiable and so can be designed into user recognition schemes. Typically, behavioural biometric modalities can be considered according to persons’ skills, style, preference, knowledge, motor-skills, or strategy while they interact with an IoT application [24]. The categories that can be derived are 1) authorship; 2) HCI; 3) indirect HCI; 4) motor skills; and 5) natural habit, based on various information extracted or gathered from a person. These categories are summarised in Fig. 4.
行为生物识别涉及可测量且可唯一识别的人类行为特征或活动模式，因此可以设计到用户识别方案中。通常，可以根据人们与物联网应用程序交互时的技能、风格、偏好、知识、运动技能或策略来考虑行为生物识别模式[24]。可以导出的类别有：1）作者身份； 2）人机交互； 3）间接人机交互； 4）运动技能； 5）自然习惯，基于从一个人提取或收集的各种信息。图 4 总结了这些类别。

• •

  Authorship-based biometrics involves verifying a person by observing peculiarities in their behavior. This includes the vocabulary used, style of writing, punctuation, or brush strokes, occuring in their writings or drawing  [31].
  基于作者身份的生物识别技术涉及通过观察一个人行为的特殊性来验证一个人。这包括他们的写作或绘画中使用的词汇、写作风格、标点符号或笔触[31]。

• •

  HCI-based biometrics, exploits a person’s inherent, distinctive, and consistent muscle actions while they use regular input devices, such as touch-devices, keyboards, computer mice, and haptics [32]. Furthermore, it leverages advanced human behavior involving knowledge, strategies, or skills exhibited by a person during interaction with smart devices.
  基于人机交互的生物识别技术利用一个人在使用常规输入设备（例如触摸设备、键盘、电脑鼠标和触觉设备）时固有的、独特的和一致的肌肉动作[32]。此外，它还利用先进的人类行为，包括人在与智能设备交互过程中表现出的知识、策略或技能。

• •

  Indirect HCI-based biometrics may be considered as an extension of the second category. It considers a person’s indirect interaction behavior, by monitoring low-level computer events (e.g., battery usage) [33], stack traces [34], application audit [35], or network traffic logs [36], or mutual interaction analysis (e.g., completely automated public Turing test to tell computers and humans apart - CAPTCHA) [37].
  基于人机交互的间接生物识别技术可以被视为第二类的延伸。它通过监视低级计算机事件（例如电池使用情况）[33]、堆栈跟踪[34]、应用程序审核[35]或网络流量日志[36]或相互交互分析来考虑人的间接交互行为（例如，完全自动化的公共图灵测试来区分计算机和人类 - CAPTCHA）[37]。

• •

  Motor-skills based behavioural biometrics can be described as the ability of a person to perform a particular action using muscle movements [38]. These muscle movements are produced as a result of coordination between the brain, skeleton, joints, and nervous system that differs from person to person [39].
  基于运动技能的行为生物识别可以描述为一个人使用肌肉运动执行特定动作的能力[38]。这些肌肉运动是大脑、骨骼、关节和神经系统之间协调的结果，因人而异[39]。

• •

  Natural habits-based biometrics constitute purely behavioural biometrics measuring persistent human behavior such as gait [40], hand-movement [41], swipe [42], grip [43], and footstep [44].
  基于自然习惯的生物识别技术构成纯粹的行为生物识别技术，测量持续的人类行为，例如步态[40]、手部运动[41]、滑动[42]、握力[43]和脚步[44]。

1. Download : Download high-res image (741KB)
   下载：下载高分辨率图像 (741KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 4. A categorization of behavioural biometrics [24].
图 4. 行为生物识别学的分类 [24]。

2.2. Sensors 2.2.传感器

IoT endpoints (devices) can provide position, orientation, or other motion-based measurements to determine unique and finite hand micro-movements. These 3-D space measurements can describe device positioning and movement while users interact. Similarly, acoustic, pressure, motion, or occupancy sensors can be used for acquiring behavioural biometric modalities such as voice, gait, or footstep for user recognition. Touch screens can be utilized to acquire touch-stroke, swipe, or touch-signature data.
物联网端点（设备）可以提供位置、方向或其他基于运动的测量，以确定独特且有限的手部微运动。这些 3D 空间测量可以描述用户交互时的设备定位和移动。类似地，声学、压力、运动或占用传感器可用于获取行为生物特征模态，例如用于用户识别的语音、步态或脚步。触摸屏可用于获取触摸笔划、滑动或触摸签名数据。

2.3. Human-to-things recognition process
2.3.人对物的识别过程

ISO2382-2017 [45] specified biometric recognition or biometrics as an automated recognition of individuals based on their biological and behavioural characteristics. ISO2382-2017 mentioned that the use of ‘authentication’ as a synonym for “biometric verification or biometric identification” is deprecated; the term biometric recognition is preferred. Thus, human-to-things recognition can be a generic term encompassing automated identification and verification of individuals in the context of IoT applications.
ISO2382-2017 [45] 将生物识别或生物识别指定为基于个人的生物和行为特征的自动识别。 ISO2382-2017提到不推荐使用“身份验证”作为“生物特征验证或生物特征识别”的同义词；优选术语“生物特征识别”。因此，人对物识别可以是一个通用术语，涵盖物联网应用背景下的个人自动识别和验证。

• •

  According to ISO2382-2017 [45], an identification process is a one-to-many comparison decision to determine whether a particular biometric data subject is in a biometric reference database. Identification systems can be employed for both negative recognition (such as preventing a single person from using multiple identities) or positive recognition for authentication purposes.
  根据ISO2382-2017 [45]，识别过程是一对多的比较决策，以确定特定的生物特征数据主体是否在生物特征参考数据库中。身份识别系统可用于消极识别（例如防止一个人使用多个身份）或用于身份验证目的的积极识别。

• •

  Similarly, ISO2382-2017 [45] defines a verification process as a comparison decision to determine the validity of a biometric claim in a verification transaction. Thus, a verification process is a one-to-one comparison in which the biometric probe(s) of a subject is compared with the biometric reference(s) of the subject to produce a comparison score. Generally, a verification system requires a labeled claimant identity as an input to be compared with the stored templates (e.g., biometrics templates) corresponding to the given label, to assert the individual’s claim. Often, verification systems are deployed for positive identification to prevent systems from zero-effort impostors and illegitimate persons.
  类似地，ISO2382-2017 [45]将验证过程定义为确定验证交易中生物特征声明有效性的比较决策。因此，验证过程是一对一的比较，其中将受试者的生物测定探针与受试者的生物测定参考进行比较以产生比较分数。一般来说，验证系统需要带标签的索赔人身份作为输入，与给定标签对应的存储模板（例如生物识别模板）进行比较，以断言个人的索赔。通常，部署验证系统是为了进行积极识别，以防止系统被零努力冒名顶替者和非法人员攻击。

2.4. Performance metrics 2.4.性能指标

3.1. Touch-strokes dynamics
3.1.触碰动态

Touch-strokes can be described as touch sequences registered by a touchscreen sensor while users navigate on touchscreen-based smart devices using their fingers [49]. Studies have shown that human musculoskeletal structure can produce finger movements that can differ from person to person [50]. Thus, a unique digital signature can be obtained from individuals’ touch-points or keystrokes collected using built-in touch sensors available in smart devices. Commonly, touch-stroke features can be categorized as spatial, timing, and motion features [51].
触摸笔划可以描述为当用户使用手指在基于触摸屏的智能设备上导航时由触摸屏传感器记录的触摸序列[49]。研究表明，人类肌肉骨骼结构可以产生因人而异的手指运动[50]。因此，可以从使用智能设备中内置触摸传感器收集的个人触摸点或击键中获得独特的数字签名。通常，触摸笔划特征可以分为空间特征、时间特征和运动特征[51]。

3.1.2. Timing features 3.1.2.计时功能

The touch-stroke timing features generation method can utilize dwell (press or hold) and flight (latency) time. Dwell time can be defined as the time duration of a touch-event of the same key and flight time can be defined as the time interval between the touch events of two successive keys. These features are directly proportional to the number of touches on the touch-screen. As an example, Fig. 5 illustrates 30 features containing 8-Type0 dwell time features and 22-Type1 to Type4 flight time features that can be extracted from the 8 touch-sequence [55].
触摸笔画时序特征生成方法可以利用停留（按下或保持）和飞行（延迟）时间。停留时间可以定义为同一按键的触摸事件的持续时间，飞行时间可以定义为两个连续按键的触摸事件之间的时间间隔。这些功能与触摸屏上的触摸次数成正比。作为一个例子，图 5 显示了 30 个特征，其中包含 8 个类型 0 的停留时间特征和 22 个类型 1 到类型 4 的飞行时间特征，这些特征可以从 8 个触摸序列中提取[55]。

1. Download : Download high-res image (506KB)
   下载：下载高分辨率图像 (506KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 5. Commonly used duration based touch-strokes timing features.
图 5. 常用的基于触摸笔画计时功能的持续时间。

The touch-stroke timing features generation method can also utilize different key-touch duration as illustrated in Fig. 6. The shortest feature-length can be termed as uni-graph, which is the timing feature extracted by taking the touch event timestamp values of the same key [56]. The timing features extracted from two, three, or more keys are termed as di-graph, tri-graph, and n-graph, respectively.
触摸笔画时序特征生成方法还可以利用不同的按键触摸持续时间，如图 6 所示。最短的特征长度可以称为单图，它是通过取触摸事件时间戳值提取的时序特征相同的键[56]。从两个、三个或更多键提取的时序特征分别称为二图、三图和n图。

1. Download : Download high-res image (132KB)
   下载：下载高分辨率图像 (132KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 6. Graph based touch-strokes timing features.
图 6. 基于图形的触摸笔画计时特征。

3.2. Swipe 3.2.滑动

Swipe can be defined as a finite touch-events sequence that occurred as a result of users touching a smart device’s touchscreen with their finger. Smart devices provide APIs to get touch coordinates, velocity, and pressure data for each touch-point [59].
滑动可以定义为用户用手指触摸智能设备触摸屏时发生的有限触摸事件序列。智能设备提供 API 来获取每个触摸点的触摸坐标、速度和压力数据 [59]。

Some of the spatial features that can be extracted from a swipe action are the touch-points timestamp, x- and y-coordinates, velocity, and acceleration. Acceleration for each touch-point can be computed mathematically, from velocity data. The touch pressure of each touch-point determines how hard the finger was pressed on the screen, and what was the touch size. Also, trajectory length, duration, average velocity, average touch-size, start and end touch coordinates can be derived from a swipe data [60], [61]. Additionally, statistical features, such as min, max, average, standard deviation, variance, kurtosis, and skewness can be computed from each 2-D touch sequence, i.e., position, velocity, acceleration, and pressure, acquired for a swipe action [62].
可以从滑动动作中提取的一些空间特征包括触摸点时间戳、x 和 y 坐标、速度和加速度。每个接触点的加速度可以根据速度数据进行数学计算。每个触摸点的触摸压力决定了手指在屏幕上按压的力度以及触摸的大小。此外，轨迹长度、持续时间、平均速度、平均触摸大小、开始和结束触摸坐标可以从滑动数据导出[60]、[61]。此外，还可以根据针对滑动动作获取的每个二维触摸序列（即位置、速度、加速度和压力）来计算统计特征，例如最小值、最大值、平均值、标准差、方差、峰度和偏度。 62]。

3.3. Touch signature 3.3.触摸签名

Touch signature, i.e., a person signing on smart devices’ touchscreen using their finger or stylus, is similar to a handwritten signature. Although, a touch signature can utilize the features that are extracted for a swipe gesture to generate a unique identification for users specified in Section 3.2.
触摸签名，即一个人使用手指或手写笔在智能设备的触摸屏上签名，类似于手写签名。尽管如此，触摸签名可以利用为滑动手势提取的特征来为第 3.2 节中指定的用户生成唯一标识。

Typically, touch signature features can be classified as global and local features [63]. Global features include total writing time, number of strokes, and signature size. Local features include local velocity, stroke angles, etc., computed at an instance of time or for a short duration. Some of the statistical features that can be extracted for touch signature are minimum, maximum, and mean of speed, acceleration, pressure, and size of the continuous strokes [64]. Further, for each stroke in a touch signature, touch-duration, segment direction, log curvature radius, stroke length to width ratio can be extracted [65], [66].
通常，触摸签名特征可以分为全局特征和局部特征[63]。全局特征包括总书写时间、笔划数和签名大小。局部特征包括在某个时刻或短时间内计算的局部速度、行程角度等。可以提取触摸签名的一些统计特征是速度、加速度、压力和连续笔画大小的最小值、最大值和平均值[64]。此外，对于触摸签名中的每个笔划，可以提取触摸持续时间、分段方向、对数曲率半径、笔划长宽比[65]、[66]。

Touch-duration can be utilized for finding similarity between touch signatures of a person. The difference between the two touch-duration sequences ($T_{difference}$) can be computed using $T_{difference}=\sum _{n=1}^N|T_s\left(n\right)-T_r\left(n\right)|$. $T_s\left(n\right)$ and $T_r\left(n\right)$ are touch-duration of $n^{th}$ touch sequence, respectively that are obtained from two touch signatures of a person. The direction (${\theta }_i$) of i-th segment having coordinates ($x_i,y_i$; $x_{i+1},y_{i+1}$) can be calculated as ${\theta }_i=arctan\left(\frac{y_{i+1}-y_i}{x_{i+1}-x_i}\right)\forall i=1toN$. After decomposing the signature into multiple strokes, Lognormal velocity distribution $v_i\left(t\right)$ of $i^{th}$ stroke for a given starting time ($t_{0i}$), stroke-length ($D_i$), logtime delay (${\mu }_i$) and logresponse time (${\sigma }_i$) can be obtained using Equation, i.e., $|v_i\left(t\right)|=\frac{D_i}{\sqrt{2\pi {\sigma }_i(t-t_{0i})}}exp(-\frac{{(ln(t-t_{0i})-{\mu }_i)}^2}{2{\sigma }_i^2})$.
触摸持续时间可用于查找人的触摸签名之间的相似性。两个触摸持续时间序列之间的差异 ( $T_{difference}$ ) 可以使用 $T_{difference}=\sum _{n=1}^N|T_s\left(n\right)-T_r\left(n\right)|$ 计算。 $T_s\left(n\right)$ 和 $T_r\left(n\right)$ 是 $n^{th}$ 触摸序列的触摸持续时间，分别是从一个人的两个触摸签名获得的。坐标为 ( $x_i,y_i$ ; $x_{i+1},y_{i+1}$ ) > ）可以计算为 ${\theta }_i=arctan\left(\frac{y_{i+1}-y_i}{x_{i+1}-x_i}\right)\forall i=1toN$ 。将签名分解为多个笔划后，给定开始时间（ $i^{th}$ 笔划的对数正态速度分布 $v_i\left(t\right)$ $t_{0i}$ ），行程长度（ $D_i$ ），对数时间延迟（ ${\mu }_i$ ）和对数响应时间（ ${\sigma }_i$ ）可以使用方程获得，即 $|v_i\left(t\right)|=\frac{D_i}{\sqrt{2\pi {\sigma }_i(t-t_{0i})}}exp(-\frac{{(ln(t-t_{0i})-{\mu }_i)}^2}{2{\sigma }_i^2})$ 。

3.4. Hand movements 3.4.手部动作

Hand movements can be defined as a finite trajectory in 3-D space for gestures like hold, upward, downward, or snap while users perform a particular activity using their smart devices. For a user’s hand-movement action, unique user-identification-signature can be generated from collected $X$, $Y$, $Z$, and $M$ coordinates. In this process, $X$, $Y$, and $Z$ streams can be collected using sensors such as Accelerometer, Gyroscope, Magnetometer, or Gravity sensors, available in smart devices. Whereas, magnitude ($M$) stream can be derived mathematically, from each sensor sample ($X$, $Y$, $Z$), i.e.,  $M=\sqrt{(X^2+Y^2+Z^2)}$.
当用户使用智能设备执行特定活动时，手部运动可以定义为 3D 空间中的有限轨迹，用于保持、向上、向下或捕捉等手势。对于用户的手势动作，可以从收集的 $X$ 、 $Y$ 、 < b4> $Z$ 和 $M$ 坐标。在此过程中， $X$ 、 $Y$ 和 $Z$ 流可以使用智能设备中提供的加速度计、陀螺仪、磁力计或重力传感器等传感器来收集。然而，幅度（ $M$ ）流可以通过数学方式从每个传感器样本（ $X$ 、 、 $Z$ ），即 $M=\sqrt{(X^2+Y^2+Z^2)}$ 。

Univariate statistical features can then be extracted from each raw stream that aid to reduce the dimensionality of raw data and improve the signal-to-noise ratio [41]. Some of the statistical features, such as min (minimum value), max (maximum value), mean (average value), standard deviation (variation from the mean value), skewness (measure of the distortion or asymmetry), kurtosis (measure of the tailedness), etc., for a dataset ($S$) containing $N$ values can be computed using Eqs. 1.
然后可以从每个原始流中提取单变量统计特征，这有助于降低原始数据的维数并提高信噪比[41]。一些统计特征，例如 min（最小值）、max（最大值）、mean（平均值）、标准差（与平均值的偏差）、偏度（扭曲或不对称的度量）、峰度（扭曲或不对称的度量）对于包含 $N$ 值的数据集 ( $S$ ) 等，可以使用等式计算： 1.(1)$\begin{array}{cc}Minimum\left(Min\right)\hfill & =\underset{i=1}{\overset{N}{\min }}{S}_i\hfill \\ Maximum\left(Max\right)\hfill & =\underset{i=1}{\overset{N}{\max }}{S}_i\hfill \\ Mean\left(\mu \right)\hfill & =\frac{1}{N}\sum _{i=1}^N{S}_i\hfill \end{array}\begin{array}{cc}StandardDeviation\left(\sigma \right)\hfill & =\sqrt{\frac{{\sum }_{i=1}^N(S_i-\mu )}{N}}\hfill \\ Kurtosis\left(k\right)\hfill & =\frac{\frac{1}{N}{\sum }_{i=1}^N{(S_i-\mu )}^4}{{\sigma }^4}\hfill \\ Skewness\left(s\right)\hfill & =\frac{\frac{1}{N}{\sum }_{i=1}^N{(S_i-\mu )}^3}{{\sigma }^3}\hfill \end{array}$

3.5. Voice 3.5.嗓音

Speech processing can be a challenging task as people have different accents, pronunciations, styles, word rates, speed of speech, speech emphasis, accent, and emotional states. Typically, a voice-based authentication system can be either text-dependent or text-independent. Fig. 7 illustrates speech processing methods encompassing speaker identification, speaker detection, and speaker verification [67].
语音处理可能是一项具有挑战性的任务，因为人们有不同的口音、发音、风格、语速、语速、语音重点、口音和情绪状态。通常，基于语音的认证系统可以是文本相关的，也可以是文本无关的。图 7 说明了包含说话人识别、说话人检测和说话人验证的语音处理方法 [67]。

1. Download : Download high-res image (381KB)
   下载：下载高分辨率图像 (381KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 7. An overview of speech processing [67].
图 7. 语音处理概述 [67]。

Voice biometrics exploit human speech parametrization or pattern matching/scoring methods to generate a unique identification signature. Human speech generation involves the lungs, vocal cords, and vocal tracts [68]. When a person speaks, the air expels from the lungs passing through the vocal cords that dilate or expand allowing the airflow to produce unvoiced or voiced sound. Subsequently, the air is resonated and reshaped by the vocal tract that consists of multiple organs such as the throat, mouth, nose, tongue, teeth, and lips. The vocal cord’s modulation, interaction, and movement of these organs can alter sound waves and produce unique sounds for each person. For a sound, the phoneme is known as the smallest distinctive unit sound of a speech [69] and pitch can be referred to as a fundamental frequency [70]. Each phoneme sound can be explained as airwaves produced by the lungs that are modulated by the vocal cords and vocal tract system.
语音生物识别技术利用人类语音参数化或模式匹配/评分方法来生成唯一的识别签名。人类言语的产生涉及肺、声带和声道[68]。当人说话时，空气从肺部排出，穿过声带，声带扩张或扩张，使气流产生清音或浊音。随后，空气通过由喉咙、口腔、鼻子、舌头、牙齿和嘴唇等多个器官组成的声道产生共鸣和重塑。声带这些器官的调制、相互作用和运动可以改变声波并为每个人产生独特的声音。对于声音来说，音素被称为语音的最小区别单位声音[69]，音高可以被称为基频[70]。每个音素声音都可以解释为由肺部产生的、由声带和声道系统调制的气波。

Speech parametrization transforms a speech signal into a set of feature vectors, such as Mel Frequency Cepstral Coefficients (MFCCs), mean Hilbert envelope coefficients (MHEC) [71], Power Normalized Cepstral Coefficients (PNCCs) [72], and non-negative matrix factorisation (NMF) [73]. MFCCs are widely used parametric features for automatic speech and speaker recognition systems [74]. A Mel is a unit of pitch [75]. The sound pairs that are perceptually equidistant in pitch are separated by an equal number of Mels. The mapping between frequency in Hertz and the Mel scale is linear below 1000 Hz and logarithmic above 1000 Hz. The Mel frequency ($mel\left(f\right)=1127ln(1+\frac{f}{700})$) can be computed from the raw acoustic frequency.
语音参数化将语音信号转换为一组特征向量，例如梅尔频率倒谱系数（MFCC）、平均希尔伯特包络系数（MHEC）[71]、功率归一化倒谱系数（PNCC）[72]和非负矩阵因式分解（NMF）[73]。 MFCC 是自动语音和说话人识别系统广泛使用的参数特征[74]。 Mel 是音高单位[75]。感知上音高等距的声音对被相同数量的梅尔分开。以赫兹为单位的频率与梅尔标度之间的映射在 1000 Hz 以下为线性，在 1000 Hz 以上为对数。梅尔频率 ( $mel\left(f\right)=1127ln(1+\frac{f}{700})$ ) 可以根据原始声频计算出来。

To extract MFCCs, first the voice signal is pre-emphasized using a first-order high-pass filter to boost the high frequencies energy. The next step involves windowing that can be performed using the Hamming function to extract spectral features from a small window of speech. Afterward, Fast Fourier Transform (FFT) is applied to extract spectral information from the windowed signal to determine the amount of energy at each frequency band. For computing MFCCs, filter banks are created with 10 filters spaced linearly below 1000 Hz, and the remaining filters spread logarithmically, above 1000 Hz collecting energy from each frequency band. After taking the log of each of the mel spectrum values. Finally, Inverse Fast Fourier Transform (IFFT) is applied extracting the energy and 12 cepstral coefficients for each frame.
为了提取 MFCC，首先使用一阶高通滤波器对语音信号进行预加重，以增强高频能量。下一步涉及可以使用汉明函数执行的加窗，以从小语音窗口中提取频谱特征。然后，应用快速傅里叶变换 (FFT) 从加窗信号中提取频谱信息，以确定每个频带的能量量。为了计算 MFCC，滤波器组由 1000 Hz 以下线性间隔的 10 个滤波器创建，其余滤波器以对数方式扩展，在 1000 Hz 以上收集来自每个频段的能量。取每个梅尔谱值的对数后。最后，应用快速傅里叶逆变换 (IFFT) 提取每帧的能量和 12 个倒谱系数。

Pattern matching/scoring methods involves probabilistic modeling (e.g., Gaussian Mixture Model (GMM) [76], Hidden Markov Models (HMMs) [77], Joint factor analysis (JFA), i-vectors [76]), template matching (e.g., vector quantization, nearest neighbor) and deep neural network trained on various combinations of i-vectors, x-vector, feature-space maximum likelihood linear regression (fMLLR) transformation [76] or Gabor filter (GF) [78]. I-vectors are low-dimensional fixed-length speaker-and-channel dependent space that is a result of joint factor analysis [79]. For extremely short utterances, i-vectors based approaches can provide an effective speaker identification solution using different scoring methods like cosine distance or probabilistic linear discriminant analysis (PLDA). In an x-vector system, DNN is trained to extract the speaker’s voice features, and the extracted speaker embedding is called x-vector [80].
模式匹配/评分方法涉及概率建模（例如高斯混合模型（GMM）[76]、隐马尔可夫模型（HMM）[77]、联合因子分析（JFA）、i-向量[76]）、模板匹配（例如、向量量化、最近邻）和深度神经网络，在 i 向量、x 向量、特征空间最大似然线性回归 (fMLLR) 变换 [76] 或 Gabor 滤波器 (GF) [78] 的各种组合上进行训练。 I 向量是低维固定长度的说话者和通道相关空间，是联合因子分析的结果[79]。对于极短的话语，基于 i 向量的方法可以使用不同的评分方法（例如余弦距离或概率线性判别分析 (PLDA)）提供有效的说话人识别解决方案。在x向量系统中，训练DNN来提取说话人的语音特征，提取的说话人嵌入称为x向量[80]。

3.6. Gait 3.6.步态

Human gait is the defined as the manner and style of walking [81]. Gait can be characterized by its cadence that is measured as the number of steps per time unit. Typically, a person’s gait varies during different activities, e.g., walking, running, hopping, ascending, or descending, etc. [82]. A gait cycle, illustrated in Fig. 8, consists of two primary phases: stance and swing [83]. The stance phase is the time-period during which feet are on the ground, constitutes approximately 60% of the gait cycle. The swing phase is the time-period during which the foot is in the air, constitutes the remaining 40% of the gait cycle. A stance phase can be further divided into 1) initial-contact and loading-response, 2) mid-contact and terminal-response, and, 3) Pre-swing. Similarly, a swing phase can be divided into 1) initial, 2) mid, and 3) terminal swing [84]. Using these parameters, both time-based and spatial features can be extracted as indicated in Table 4.
人类步态被定义为行走的方式和风格[81]。步态可以通过节奏来表征，节奏以每个时间单位的步数来衡量。通常，一个人的步态在不同的活动中会有所不同，例如步行、跑步、跳跃、上升或下降等[82]。如图 8 所示，步态周期由两个主要阶段组成：站立和摆动 [83]。站立阶段是脚接触地面的时间段，约占步态周期的 60%。摆动阶段是脚在空中的时间段，构成步态周期的剩余 40%。站立阶段可进一步分为 1) 初始接触和负载响应，2) 中间接触和最终响应，以及 3) 预摆动。类似地，摆动阶段可以分为 1）初始，2）中期和 3）末端摆动 [84]。使用这些参数，可以提取基于时间和空间的特征，如表 4 所示。

1. Download : Download high-res image (368KB)
   下载：下载高分辨率图像 (368KB)
2. Download : Download full-size image
   下载：下载全尺寸图像

Fig. 8. An illustration of a gait cycle.
图 8.步态周期的图示。

Table 4. Gait features. 表 4.步态特征。

#	Spatial 空间	Time 时间
1.	Stride length (cm) 步幅（厘米）	Duration of step (milli sec) 步骤持续时间（毫秒）
2.	Step length (cm) 步长（厘米）	Stride duration (milli sec) 步幅持续时间（毫秒）
3.	Stride width or base of support (cm) 步幅宽度或支撑基础（厘米）	Stance phase (milli sec) 站立阶段（毫秒）
4.	Internal/External Angle ($\mathrm{deg}$) 内/外角 ( $\mathrm{deg}$ )	Swing phase (milli sec) 摆动相位（毫秒）
5.	Speed (m/s or cm/s) 速度（米/秒或厘米/秒）	Cadence(steps/min) 步频（步/分钟）
6.	Walk ratio (cm/step/min) 步行比（厘米/步/分钟）	-

Some more gait features [40] that can be analyzed for user recognition are gait variability and angular kinematics. Gait Variability (GV) can be defined as changes in gait parameters from one stride to the next. In a gait cycle, the coefficient of variation (CV) that is a measure of total variability can be calculated as root mean square (RMS) of standard deviation ($\sigma$) of the moment over stride period $t$ mean of the absolute moment of force over stride period using Equation, i.e., $CV=\frac{\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\sigma }^2}}{\frac{1}{n}{\sum }_{i=1}^n\left|X_i\right|}$.
可以分析用户识别的更多步态特征 [40] 包括步态变异性和角度运动学。步态变异性 (GV) 可以定义为步态参数从一步到下一步的变化。在一个步态周期中，作为总变异性度量的变异系数 (CV) 可以计算为标准差 ( $\sigma$ ) 的均方根 (RMS)步幅周期内的力矩 $t$ 使用方程计算步幅周期内绝对力力矩的平均值，即 $CV=\frac{\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\sigma }^2}}{\frac{1}{n}{\sum }_{i=1}^n\left|X_i\right|}$ 。

Angular Kinematics of joint angles refers to the kinematics analysis of angular motion [40]. Angular displacement (the difference between the initial and final angular position), angular velocity (change in angular position over a period of time), and angular acceleration (change in angular velocity over a period of time) can be obtained using Eqs. 2.
关节角度的角运动学是指角运动的运动学分析[40]。角位移（初始角位置与最终角位置之差）、角速度（一段时间内角位置的变化）和角加速度（一段时间内角速度的变化）可以使用等式获得。 2.(2)$\begin{array}{cc}Angulardisplacement\left(\Delta \theta \right)\hfill & ={\theta }_{final}-{\theta }_{initial}\hfill \\ angularvelocity\left(\omega \right)\hfill & =\frac{d\theta }{dt}\hfill \\ angularacceleration\left(\alpha \right)\hfill & =\frac{d\omega }{dt}\hfill \end{array}$

3.7. Footstep 3.7.脚步

A footstep is defined as a combination of a single left and right stride of a person. Footstep features include stride length, stride direction, timing information, acoustic and psycho-acoustic parameters, spatial positions, and relative pressure values in foot regions. These features can be captured using a range of sensors including floor-based sensors[85], such as piezoelectric sensors, switch sensors, or fabric-based pressure mapping sensors.
脚步被定义为一个人向左和向右迈出的单个步幅的组合。脚步特征包括步长、步幅方向、计时信息、声学和心理声学参数、空间位置以及足部区域的相对压力值。这些特征可以使用一系列传感器来捕获，包括基于地板的传感器[85]，例如压电传感器、开关传感器或基于织物的压力映射传感器。

Ground Reaction Force (GRF) is the common feature providing a description of a person’s footstep force acquired from pressure sensors [44]. Ground Reaction Force ($GR{F}_i$) per sensor can be computed by accumulating each $i^{th}$ sensor pressure amplitude from time $t=1$ to $t=T_{\max }$ using Equation, i.e., $GR{F}_i={\sum }_{t=1}^{T_{\max }}{P}_i\left[t\right]$.
地面反作用力（GRF）是描述从压力传感器获取的人的脚步力的常见特征[44]。每个传感器的地面反作用力 ( $GR{F}_i$ ) 可以通过从时间 $i^{th}$ 传感器压力幅度来计算/b4> $t=1$ 到 $t=T_{\max }$ 使用方程，即 $GR{F}_i={\sum }_{t=1}^{T_{\max }}{P}_i\left[t\right]$ 。

Furthermore, using Eq. 3 time-series arrays, namely, average spatial pressure ($S{P}_{ave}$), cumulative spatial pressure ($S{P}_{cumulative}$), upper ($S{P}_{upper}$) and lower ($S{P}_{lower}$) contours can be generated from the pressure signals acquired from $N$ sensors for a $T$ time-period [86].
此外，使用方程。 3个时间序列数组，分别为平均空间压力( $S{P}_{ave}$ )、累积空间压力( $S{P}_{cumulative}$ )、上限( $S{P}_{upper}$ ）和下部（ $S{P}_{lower}$ ）轮廓可以根据从 获取的压力信号生成 $N$ 传感器用于 $T$ 时间段[86]。(3)$\begin{array}{cccc}S{P}_{ave}\left[t\right]\hfill & =\sum _{i=1}^N{P}_i\left[t\right]\hfill & S{P}_{cumulative}\left[t\right]\hfill & =\sum _{i=1}^N{P}_i\left[t\right]+\sum _{i=1}^N{P}_i[t-1]\hfill \\ S{P}_{upper}\left[t\right]\hfill & =\underset{i=1}{\overset{N}{max}}{S}_i\left[t\right]\hfill & S{P}_{lower}\left[t\right]\hfill & =\underset{i=1}{\overset{N}{min}}{S}_i\left[t\right]\hfill \end{array}$where, $P_i\left[t\right]$ is the differential pressure value from the $i^{th}$ sensors at the time $t$, and, $N$ is the total number of sensors. Footstep analysis is applicable for numerous applications, such as predicting human action, security, and surveillance at public places [86].
其中， $P_i\left[t\right]$ 是 $i^{th}$ 传感器在 $t$ ， $N$ 是传感器总数。足迹分析适用于许多应用，例如预测人类行为、公共场所的安全和监视[86]。