A survey of human-computer interaction (HCI) & natural habits-based behavioural biometric modalities for user recognition schemes
针对用户识别方案的人机交互 (HCI) 和基于自然习惯的行为生物识别模式的调查

计算机科学TOPEI检索SCI升级版 计算机科学1区SCI基础版 工程技术2区IF 8.0 如果8.0SWJTU A++ 西南交通大学A++SWUFE A 西南财经大学A
https://doi.org/10.1016/j.patcog.2023.109453 Get rights and content 获取权利和内容

Highlights 强调

  • The article presents a survey of the human-computer interaction and natural habits-based biometrics, namely, touchstroke, swipe, touch-signature, hand-movements, voice, gait, and single footstep that can be acquired from smart devices equipped with motion sensors, touch screens, and microphones or by external IoT sensors or nodes in an unobtrusive manner.
    本文对人机交互和基于自然习惯的生物识别进行了调查,即可以从配备运动传感器的智能设备获取的触摸、滑动、触摸签名、手部动作、语音、步态和单脚步。 、触摸屏和麦克风或通过外部物联网传感器或节点以不显眼的方式。

  • The article elicits attributes and features of the aforementioned behavioral biometrics that can be exploited for designing reliable user recognition schemes. We discuss the methodologies, classifiers, datasets, and performance results of recent user recognition schemes that employ these behavioral biometrics modalities.

  • The article presents security, privacy, and usability attributes with regard to the (CIA) properties in human-to-things recognition schemes.
    本文介绍了人对物识别方案中 (CIA) 属性的安全性、隐私性和可用性属性。

  • The article discusses challenges, limitations, prospects, and opportunities associated with behavioral biometric-based user recognition schemes. The prospects and market trends indicate that behavioral biometrics can instigate innovative ways to implement implicit (frictionless), continuous (active), or risk-based (non-static) recognition schemes for IoT applications.

  • Ultimately, with the availability of smart sensors, advanced machine learning algorithms, and powerful IoT platforms, behavioral biometrics can substitute conventional recognition schemes, thus, reshaping the existing user recognition landscape.

Abstract 抽象的

The proliferation of Internet of Things (IoT) systems is having a profound impact across all aspects of life. Recognising and identifying particular users is central to delivering the personalised experience that citizens want to experience, and that organisations wish to deliver. This article presents a survey of human-computer interaction-based (HCI-based) and natural habits-based behavioural biometrics that can be acquired unobtrusively through smart devices or IoT sensors for user recognition purposes. Robust and usable user recognition is also a security requirement for emerging IoT ecosystems to protect them from adversaries. Typically, it can be specified as a fundamental building block for most types of human-to-things accountability principles and access-control methods. However, end-users are facing numerous security and usability challenges in using currently available knowledge- and token-based recognition (i.e., authentication and identification) schemes. To address the limitations of conventional recognition schemes, biometrics, naturally come as a first choice to supporting sophisticated user recognition solutions. We perform a comprehensive review of touch-stroke, swipe, touch signature, hand-movements, voice, gait and footstep behavioural biometrics modalities. This survey analyzes the recent state-of-the-art research of these behavioural biometrics with a goal to identify their attributes and features for generating unique identification signatures. Finally, we present security, privacy, and usability evaluations that can strengthen the designing of robust and usable user recognition schemes for IoT applications.
物联网 (IoT) 系统的激增正在对生活的各个方面产生深远的影响。识别和识别特定用户对于提供公民想要体验的个性化体验以及组织希望提供的个性化体验至关重要。本文介绍了基于人机交互(基于 HCI)和基于自然习惯的行为生物识别技术的调查,这些生物识别技术可以通过智能设备或物联网传感器以不显眼的方式获取,以实现用户识别目的。强大且可用的用户识别也是新兴物联网生态系统的安全要求,以保护它们免受对手的攻击。通常,它可以被指定为大多数类型的人对物问责原则和访问控制方法的基本构建块。然而,最终用户在使用当前可用的基于知识和基于令牌的识别(即身份验证和识别)方案时面临着许多安全性和可用性挑战。为了解决传统识别方案的局限性,生物识别技术自然成为支持复杂的用户识别解决方案的首选。我们对触摸、滑动、触摸签名、手部动作、声音、步态和脚步行为生物识别模式进行全面审查。这项调查分析了这些行为生物识别技术的最新研究成果,旨在识别它们的属性和特征,以生成独特的识别签名。最后,我们提出了安全性、隐私性和可用性评估,可以加强物联网应用程序稳健且可用的用户识别方案的设计。

Keywords 关键词

Internet of Things (IoT)
User recognition
Behavioural biometrics

物联网 (IoT)用户识别行为生物识别安全隐私可用性

1. Introduction 一、简介

IoT ecosystems, integrating smart sensors, actuators, advanced communications, efficient computation, and artificial intelligence, have the power to transform the way we live and work. Almost every business vertical has started to embrace IoT technology [1]. This includes sectors as diverse as automotive, energy, entertainment, education, food, finance, healthcare, and transportation where smart integrated systems are delivering improved quality of life and resource efficiency by providing security-sensitive services via IoT applications. Bera et al. [2] reported that user authentication, access control, key management, and intrusion detection are essential requirements to prevent real-time data access directly from the IoT-enabled smart devices that are deployed in IoT ecosystems. Studies have indicated that application-layer attacks in the IoT are particularly complex to detect and deflect [3], [4]. Ultimately, any security breach of IoT ecosystems has the potential for profound consequences on consumers and society [5]. Therefore, robust and usable Authentication, Authorization and Accounting (AAA) mechanisms for applications bridging humans and IoT ecosystems, which can be specified as IoT Applications, are critical for maintaining confidentiality, integrity, availability (CIA) in the system illustrated in Fig. 1.
物联网生态系统集成了智能传感器、执行器、先进通信、高效计算和人工智能,有能力改变我们的生活和工作方式。几乎每个垂直行业都开始采用物联网技术[1]。这包括汽车、能源、娱乐、教育、食品、金融、医疗保健和交通等各个领域,其中智能集成系统通过物联网应用提供安全敏感的服务,从而提高生活质量和资源效率。贝拉等人。 [2] 报告称,用户身份验证、访问控制、密钥管理和入侵检测是防止直接从部署在物联网生态系统中的支持物联网的智能设备进行实时数据访问的基本要求。研究表明,物联网中的应用层攻击的检测和转移特别复杂[3]、[4]。最终,物联网生态系统的任何安全漏洞都可能对消费者和社会产生深远的影响[5]。因此,为连接人类和物联网生态系统的应用程序(可指定为物联网应用程序)提供强大且可用的身份验证、授权和计费 (AAA) 机制,对于维护图 1 所示系统中的机密性、完整性、可用性 (CIA) 至关重要。

Fig. 1
  1. Download : Download high-res image (140KB)
    下载:下载高分辨率图像 (140KB)
  2. Download : Download full-size image

Fig. 1. AAA mechanisms vs. CIA properties in the context of IoT applications.
图 1.物联网应用背景下的 AAA 机制与 CIA 属性。

Many IoT ecosystems still rely on traditional Personal Identification Numbers (PINs), passwords, and tokens based user recognition mechanisms [6]. This is despite, users facing both security and usability challenges in using these conventional (knowledge- and token-based) recognition schemes [7], [8]. Further, the decision process in conventional authentication mechanisms is usually binary [9]. PINs and passwords can be easily guessed, shared, cloned, or stolen [10]. Conventional authentication schemes are also prone to a wide range of common attacks [11], such as dictionary-, observation- and replay-attacks. Weak passwords remain the major cause of botnet-based attacks, such as Mirai, on huge numbers of IoT systems [12]. Additionally, they possess several usability issues [13], such as placing overwhelming cognitive load on users and ergonomic inefficiencies for newer IoT end-points. As such, human-to-things recognition schemes for IoT ecosystems require rethinking, with behavioural biometrics providing an appropriate alternative to overcoming the drawbacks present in conventional authentication schemes.
许多物联网生态系统仍然依赖传统的个人识别码 (PIN)、密码和基于令牌的用户识别机制 [6]。尽管如此,用户在使用这些传统的(基于知识和令牌的)识别方案时面临安全性和可用性挑战[7],[8]。此外,传统认证机制中的决策过程通常是二元的[9]。 PIN 和密码很容易被猜出、共享、克隆或窃取 [10]。传统的身份验证方案还容易受到各种常见攻击[11],例如字典攻击、观察攻击和重放攻击。弱密码仍然是大量物联网系统上基于僵尸网络的攻击(例如 Mirai)的主要原因 [12]。此外,它们还存在一些可用性问题[13],例如给用户带来巨大的认知负担以及较新的物联网端点的人体工程学效率低下。因此,物联网生态系统的人对物识别方案需要重新思考,行为生物识别技术可以提供适当的替代方案来克服传统身份验证方案中存在的缺点。

This article presents a comprehensive review of touch-stroke, swipe, touch signature, hand-movements, voice, gait and footstep behavioural biometric modalities (refer Fig. 2) for designing user recognition schemes in emerging IoT ecosystems. The motivation for this particular selection of modalities is provided by the current focus of academic research, and the industrial trend towards human-computer interaction (HCI) and natural habits-based behavioural biometrics-based recognition schemes. For instance, ViewSonic and Namirial partnered to deliver a behavioural biometric eSignature solution that includes the behavioural biometric of handwritten signatures to boost electronic signature security and reliability [14].
本文全面回顾了触摸笔画、滑动、触摸签名、手部动作、语音、步态和脚步行为生物识别模式(参见图 2),用于在新兴物联网生态系统中设计用户识别方案。这种特定模式选择的动机是由当前学术研究的焦点以及人机交互 (HCI) 和基于自然习惯的行为生物识别识别方案的行业趋势提供的。例如,ViewSonic 和 Namirial 合作提供了行为生物识别电子签名解决方案,其中包括手写签名的行为生物识别,以提高电子签名的安全性和可靠性 [14]。

Fig. 2
  1. Download : Download high-res image (596KB)
    下载:下载高分辨率图像 (596KB)
  2. Download : Download full-size image

Fig. 2. Use of behavioural biometric for Authentication, Authorization and Accounting (AAA) mechanisms for IoT applications (Source: Google Images).
图 2. 使用行为生物识别技术进行物联网应用的身份验证、授权和计费 (AAA) 机制(来源:Google 图片)。

Banking sectors are investigating characteristics including touch-stroke dynamics to generate a trusted user profiles for distinguishing between normal and unusual user behavior, as a means to detect fraudulent users [15]. For example, leading companies, such as BehavioSec [16] and BioCatch [17] are leveraging behavioural biometrics, including swipe or touch gestures, typing rhythm, or the particular way an individual holds their device, to offer enterprise-scale security solutions for continual and risk-based authentication or fraud detection. Also, many electronic payment card providers are investigating behavioural biometrics for cutting-edge payment systems of the future [18].
银行部门正在研究包括触摸动态在内的特征,以生成可信的用户配置文件,以区分正常和异常的用户行为,作为检测欺诈用户的手段[15]。例如,BehavioSec [16] 和 BioCatch [17] 等领先公司正在利用行为生物识别技术(包括滑动或触摸手势、打字节奏或个人握持设备的特定方式)来提供企业级安全解决方案,以实现持续的安全解决方案。基于风险的身份验证或欺诈检测。此外,许多电子支付卡提供商正在研究未来尖端支付系统的行为生物识别技术[18]。

A study of biometrics to achieve intelligent, convenient, and secure solutions for smart cities and smart transportation are presented in [19] and [20], respectively. Sensor-based activity recognition [21], such as gait, can be used to verify commuters through their walking patterns, thereby replacing the need for a travel pass to access public transportation. NEC Corporation and SITA have collaborated to roll out a walk-through, contactless digital identity solution for airports leveraging their biometric identity management platform to facilitate a non-intrusive method of identity verification [22]. So large is the potential that the market study forecasts that by 2025 behavioural biometrics market will reach 3.92 Billion [23].
[19]和[20]分别介绍了生物识别技术的研究,以实现智慧城市和智能交通的智能、便捷和安全的解决方案。基于传感器的活动识别[21],例如步态,可用于通过步行模式验证通勤者,从而取代旅行通行证来使用公共交通。 NEC 公司和 SITA 合作,利用其生物识别身份管理平台,为机场推出了步行式非接触式数字身份解决方案,以促进非侵入式身份验证方法 [22]。潜力如此之大,市场研究预测,到 2025 年,行为生物识别市场将达到 39.2 亿[23]。

1.1. Objectives and survey strategy

The objective of this article is to survey HCI and natural habits-based biometrics that can be utilized by researchers and engineers to design uni-modal or multi-modal user recognition schemes (leveraging concepts such as implicit, continuous, or risk-based  [9]) for security-sensitive applications, thus, safeguarding IoT ecosystems. Fig. 3 illustrates the timeline and Table 1 lists previous surveys related to the behavioural biometric modalities covered in this article.
本文的目的是调查人机交互和基于自然习惯的生物识别技术,研究人员和工程师可以利用这些生物识别技术来设计单模式或多模式用户识别方案(利用隐式、连续或基于风险等概念[9] ])适用于安全敏感的应用程序,从而保护物联网生态系统。图 3 说明了时间线,表 1 列出了之前与本文涵盖的行为生物识别模式相关的调查。

Fig. 3
  1. Download : Download high-res image (965KB)
    下载:下载高分辨率图像 (965KB)
  2. Download : Download full-size image

Fig. 3. Timeline showing the important events related to behavioural biometric modalities covered in this article.
图 3. 时间线显示了与本文涵盖的行为生物识别模式相关的重要事件。

Table 1. Earlier behavioural biometrics surveys.
表 1. 早期的行为生物识别调查。

RefYear Contributions 贡献
Yampolskiy & Govindaraju [24]
扬波尔斯基和戈文达拉茹 [24]
2008This survey presented a classification of behavioural biometrics based on skills, style, preference, knowledge, motor skills, or strategy applied by humans.
Meng et al. [25] 孟等人。 [25]2015This survey covered the development of biometric user authentication techniques on mobile phones. And, presented a study of voice, signature, gait, behavior profiling, keystroke and touch dynamics behavioural biometrics.
Alzubaidi & Kalita [26] 阿尔祖拜迪和卡利塔 [26]2016This survey investigated authentication of smartphone users based on handwaving, gait, touchscreen, keystroke, voice, signature and general profiling behavioural biometrics.
Oak [27] 橡木 [27]2018This survey analyzed persons’ behavior, such as keystroke dynamics, mouse dynamics, haptics, gait, and log files, for their designing persistent security solutions.
Dang et al. [28] 党等人。 [28]2020This survey focused on Human activity recognition (HAR) for designing context-aware applications for emerging domains like IoT and healthcare by analyzing sensor- and vision-based behavioural patterns.
这项调查的重点是人类活动识别 (HAR),通过分析基于传感器和视觉的行为模式,为物联网和医疗保健等新兴领域设计上下文感知应用程序。
Stylios et al. [29] 斯蒂利奥斯等人。 [29]2020This survey presented the classification of behavioural biometrics technologies. It reviewed behavioural traits like gait, touch gestures, keystroke dynamics, hand-waving, behavioural profile, power consumption, for continuous authentication for mobile devices.

In this survey, we first elucidate attributes and features of behavioural biometric modalities that can be acquired from smart devices equipped with motion sensors, touch screens, and microphones or by external IoT sensors or nodes in an unobtrusive manner. We discuss the methodologies, classifiers, datasets, and performance results of recent user recognition schemes that employ these behavioural biometrics modalities. We then present security, privacy, and usability attributes with regard to the CIA properties in human-to-things recognition schemes. Ultimately, the challenges, limitations, prospects, and opportunities associated with behavioural biometric-based user recognition schemes are presented.
在本次调查中,我们首先阐明行为生物识别模式的属性和特征,这些属性和特征可以从配备运动传感器、触摸屏和麦克风的智能设备或通过外部物联网传感器或节点以不显眼的方式获取。我们讨论了采用这些行为生物识别模式的最新用户识别方案的方法、分类器、数据集和性能结果。然后,我们介绍人对物识别方案中 CIA 属性的安全性、隐私性和可用性属性。最后,提出了与基于行为生物识别的用户识别方案相关的挑战、局限性、前景和机遇。

1.2. Article structure 1.2.文章结构

The article is structured as follows: Section 2 discusses behavioural biometrics, sensors, human-to-things recognition mechanisms and performance metrics. Section 2.1 elicits attributes and features of touch-stroke, swipe, touch signature, hand-movements, voice, gait, and footstep modalities that can be exploited for designing user recognition schemes. Section 4 presents the state-of-the-arts of user recognition schemes based on modalities discussed in Section 2.1. Section 5 presents a discussion on security, privacy, and usability of behavioural biometric-based user recognition schemes. Section 6 discusses the open challenges and limitations that deserve attention together with prospects and opportunities for evolving and designing behavioural biometric-based human-to-things recognition schemes. Section 7 concludes the article.
本文的结构如下:第 2 部分讨论行为生物识别、传感器、人对物识别机制和性能指标。第 2.1 节引出了触摸、滑动、触摸签名、手部动作、语音、步态和脚步模式的属性和特征,可用于设计用户识别方案。第 4 节介绍了基于第 2.1 节中讨论的模式的最先进的用户识别方案。第 5 节讨论了基于行为生物识别的用户识别方案的安全性、隐私性和可用性。第 6 节讨论了值得关注的开放挑战和限制,以及发展和设计基于行为生物识别的人对物识别方案的前景和机遇。第 7 节总结了本文。

2. Background 2. 背景

Despite many advancements in recent years, human-to-things recognition (identification and authentication) remains a challenge for emerging IoT ecosystems [30]. Evidently, with improvements in sensors technology, the opportunity to evolve behavioural biometric-based human-to-things recognition schemes has increased significantly.

2.1. Behavioural biometrics

Behavioural biometrics involve human behavioural characteristics or activity patterns that are measurable and uniquely identifiable and so can be designed into user recognition schemes. Typically, behavioural biometric modalities can be considered according to persons’ skills, style, preference, knowledge, motor-skills, or strategy while they interact with an IoT application [24]. The categories that can be derived are 1) authorship; 2) HCI; 3) indirect HCI; 4) motor skills; and 5) natural habit, based on various information extracted or gathered from a person. These categories are summarised in Fig. 4.
行为生物识别涉及可测量且可唯一识别的人类行为特征或活动模式,因此可以设计到用户识别方案中。通常,可以根据人们与物联网应用程序交互时的技能、风格、偏好、知识、运动技能或策略来考虑行为生物识别模式[24]。可以导出的类别有:1)作者身份; 2)人机交互; 3)间接人机交互; 4)运动技能; 5)自然习惯,基于从一个人提取或收集的各种信息。图 4 总结了这些类别。

  • Authorship-based biometrics involves verifying a person by observing peculiarities in their behavior. This includes the vocabulary used, style of writing, punctuation, or brush strokes, occuring in their writings or drawing  [31].

  • HCI-based biometrics, exploits a person’s inherent, distinctive, and consistent muscle actions while they use regular input devices, such as touch-devices, keyboards, computer mice, and haptics [32]. Furthermore, it leverages advanced human behavior involving knowledge, strategies, or skills exhibited by a person during interaction with smart devices.

  • Indirect HCI-based biometrics may be considered as an extension of the second category. It considers a person’s indirect interaction behavior, by monitoring low-level computer events (e.g., battery usage) [33], stack traces [34], application audit [35], or network traffic logs [36], or mutual interaction analysis (e.g., completely automated public Turing test to tell computers and humans apart - CAPTCHA[37].
    基于人机交互的间接生物识别技术可以被视为第二类的延伸。它通过监视低级计算机事件(例如电池使用情况)[33]、堆栈跟踪[34]、应用程序审核[35]或网络流量日志[36]或相互交互分析来考虑人的间接交互行为(例如,完全自动化的公共图灵测试来区分计算机和人类 - CAPTCHA)[37]。

  • Motor-skills based behavioural biometrics can be described as the ability of a person to perform a particular action using muscle movements [38]. These muscle movements are produced as a result of coordination between the brain, skeleton, joints, and nervous system that differs from person to person [39].

  • Natural habits-based biometrics constitute purely behavioural biometrics measuring persistent human behavior such as gait [40], hand-movement [41], swipe [42], grip [43], and footstep [44].

Fig. 4
  1. Download : Download high-res image (741KB)
    下载:下载高分辨率图像 (741KB)
  2. Download : Download full-size image

Fig. 4. A categorization of behavioural biometrics [24].
图 4. 行为生物识别学的分类 [24]。

2.2. Sensors 2.2.传感器

The rapid evolution of system-on-chip (SoC) and wireless technologies play a vital role in evolving smarter, smaller, accurate, and efficient sensors for behavioural biometric data acquisition. Table 2 describes sensors that can be integrated into smart devices and portable IoT devices for acquiring behavioural biometric modalities covered in Section 2.1.
片上系统 (SoC) 和无线技术的快速发展在发展更智能、更小、更准确和更高效的行为生物识别数据采集传感器方面发挥着至关重要的作用。表 2 介绍了可集成到智能设备和便携式 IoT 设备中的传感器,用于获取第 2.1 节中所述的行为生物识别模式。

Table 2. Sensors for acquiring behavioural biometric modalities.
表 2. 用于获取行为生物识别模式的传感器。

Category 类别Sensor description 传感器说明Sensor Type 传感器类型
Position 位置Position sensors can be linear, angular, or multi-axis. It measures the position of an object that can be either relative in terms of displacements or absolute positions.
Proximity sensor, Potentiometer, Inclinometer
Motion, Occupancy 运动、占用Motion and occupancy sensors detect movement and presence of people and objects, respectively.
Electric eye, RADAR, Depth Camera
Velocity, Acceleration, Direction
Velocity sensors can be linear or angular. It measures the rate of change linear or angular displacement. Acceleration sensors measure the rate of change of velocity. Magnetometer estimates the device orientation relative to earth’s magnetic north. Gravity sensor indicates the direction and magnitude of gravity.
Accelerometer, Gyroscope, Magnetometer, Gravity sensor
Pressure 压力Pressure sensors detect force per unit area
Barometer, bourdon gauge, piezometer
Force 力量Force sensors detect resistance changes when a force, pressure, or mechanical stress is applied.
Force gauge, Viscometer, Tactile sensor (Touch sensor), Capacitive touchscreen
Acoustic, Voice 声学、语音Acoustic sensors measure sound levels transform it into digital or analog data signals.
Microphone, geophone, hydrophone

IoT endpoints (devices) can provide position, orientation, or other motion-based measurements to determine unique and finite hand micro-movements. These 3-D space measurements can describe device positioning and movement while users interact. Similarly, acoustic, pressure, motion, or occupancy sensors can be used for acquiring behavioural biometric modalities such as voice, gait, or footstep for user recognition. Touch screens can be utilized to acquire touch-stroke, swipe, or touch-signature data.
物联网端点(设备)可以提供位置、方向或其他基于运动的测量,以确定独特且有限的手部微运动。这些 3D 空间测量可以描述用户交互时的设备定位和移动。类似地,声学、压力、运动或占用传感器可用于获取行为生物特征模态,例如用于用户识别的语音、步态或脚步。触摸屏可用于获取触摸笔划、滑动或触摸签名数据。

2.3. Human-to-things recognition process

ISO2382-2017 [45] specified biometric recognition or biometrics as an automated recognition of individuals based on their biological and behavioural characteristics. ISO2382-2017 mentioned that the use of ‘authentication’ as a synonym for “biometric verification or biometric identification” is deprecated; the term biometric recognition is preferred. Thus, human-to-things recognition can be a generic term encompassing automated identification and verification of individuals in the context of IoT applications.
ISO2382-2017 [45] 将生物识别或生物识别指定为基于个人的生物和行为特征的自动识别。 ISO2382-2017提到不推荐使用“身份验证”作为“生物特征验证或生物特征识别”的同义词;优选术语“生物特征识别”。因此,人对物识别可以是一个通用术语,涵盖物联网应用背景下的个人自动识别和验证。

  • According to ISO2382-2017 [45], an identification process is a one-to-many comparison decision to determine whether a particular biometric data subject is in a biometric reference database. Identification systems can be employed for both negative recognition (such as preventing a single person from using multiple identities) or positive recognition for authentication purposes.
    根据ISO2382-2017 [45],识别过程是一对多的比较决策,以确定特定的生物特征数据主体是否在生物特征参考数据库中。身份识别系统可用于消极识别(例如防止一个人使用多个身份)或用于身份验证目的的积极识别。

  • Similarly, ISO2382-2017 [45] defines a verification process as a comparison decision to determine the validity of a biometric claim in a verification transaction. Thus, a verification process is a one-to-one comparison in which the biometric probe(s) of a subject is compared with the biometric reference(s) of the subject to produce a comparison score. Generally, a verification system requires a labeled claimant identity as an input to be compared with the stored templates (e.g., biometrics templates) corresponding to the given label, to assert the individual’s claim. Often, verification systems are deployed for positive identification to prevent systems from zero-effort impostors and illegitimate persons.
    类似地,ISO2382-2017 [45]将验证过程定义为确定验证交易中生物特征声明有效性的比较决策。因此,验证过程是一对一的比较,其中将受试者的生物测定探针与受试者的生物测定参考进行比较以产生比较分数。一般来说,验证系统需要带标签的索赔人身份作为输入,与给定标签对应的存储模板(例如生物识别模板)进行比较,以断言个人的索赔。通常,部署验证系统是为了进行积极识别,以防止系统被零努力冒名顶替者和非法人员攻击。

2.4. Performance metrics 2.4.性能指标

In a biometric system designed to distinguish between a legitimate user or an impostor, there can be four possible scenarios. These are derived from the person being legitimate or not, and being (correctly or incorrectly) identified as legitimate or not. These are termed true acceptance (TA) or false rejection (FR) and true rejection (TR) or false acceptance (FA[46]. Table 3 describe the most commonly used indicators for the performance evaluation of biometric systems.
在旨在区分合法用户或冒名顶替者的生物识别系统中,可能有四种可能的情况。这些源自该人是否合法,以及被(正确或错误)识别为合法或不合法。这些被称为真正接受( TA )或错误拒绝( FR )和真正拒绝( TR )或错误接受( FA )[46]。表 3 描述了生物识别系统性能评估最常用的指标。

Table 3. Performance metrics for biometric systems.
表 3. 生物识别系统的性能指标。

Indicator 指标Description 描述
True Acceptance Rate (TAR)
真实接受率 (TAR)
This is the ratio of TA legitimate user attempts to the overall number of attempts (TA+FR). A higher TAR indicates that the system performs better in recognizing a legitimate user.
这是 TA 合法用户尝试与总尝试次数 ( TA+FR ) 的比率。 TAR 越高,表明系统在识别合法用户方面表现更好。
False Rejection Rate (FRR)
错误拒绝率 (FRR)
This is the ratio of FR legitimate user attempts to the overall attempts (TA+FR). FRR is a complement of TAR and it can be calculated as FRR = 1 - TAR. ISO/IEC 19795-1:2006 [47] also denote the term FRR as False Non-Match Rate (FNMR).
这是 FR 合法用户尝试与总体尝试 ( TA+FR ) 的比率。 FRR 是 TAR 的补数,计算公式为 FRR = 1 - TAR。 ISO/IEC 19795-1:2006 [47] 还将术语 FRR 表示为错误不匹配率 (FNMR)。
False Acceptance Rate (FAR)
错误接受率 (FAR)
This is the ratio of FA impostor attempts to overall attempts (FA+TR). A lower FAR means the system is robust to impostor attempts. ISO/IEC 19795-1:2006 [47] also specified the term FAR as False Match Rate (FMR).
这是 FA 冒充者尝试与总体尝试的比率 ( FA+TR )。 FAR 较低意味着系统对于冒充者的尝试具有鲁棒性。 ISO/IEC 19795-1:2006 [47] 还将术语 FAR 指定为错误匹配率 (FMR)。
True Rejection Rate (TRR)
真实拒绝率 (TRR)
This is the ratio of TR attempts of impostors to all overall attempts (FA+TR). TRR is the complement of FAR and can be calculated as TRR = 1 - FAR.
这是冒充者的 TR 尝试与所有总体尝试的比率 ( FA+TR )。 TRR 是 FAR 的补数,计算公式为 TRR = 1 - FAR。
Equal Error Rate (EER) 等错误率 (EER)It is the value where both errors rates, FAR and FRR, are equal (i.e., FAR = FRR).
它是错误率 FAR 和 FRR 相等时的值(即 FAR = FRR)。
Half Total Error Rate (HTER)
总错误率的一半 (HTER)
It is the average of FAR and FRR [48]. HTER and EER are identical for a given threshold with a weight set to 0.5 except that HTER can be use for measuring a classifier’s performance.
它是 FAR 和 FRR 的平均值[48]。对于权重设置为 0.5 的给定阈值,HTER 和 EER 是相同的,只是 HTER 可用于测量分类器的性能。
Accuracy 准确性The ratio of (TA+TR) to (TA+FR+TR+FA).
( TA+TR ) 与 ( TA+FR+TR+FA ) 的比率。
Receiver-Operating Characteristic (ROC)
接收器工作特性 (ROC)
ROC plot is a visual characterization of trade-off between FAR and TAR [47]. In simple terms, this is a plot between correctly raised alarms against incorrectly raised alarm. The curve is generated by plotting the FAR versus the TAR for varying thresholds to assess the classifier’s performance.
ROC 图是 FAR 和 TAR 之间权衡的直观表征[47]。简单来说,这是正确发出警报与错误发出警报之间的关系图。该曲线是通过绘制不同阈值的 FAR 与 TAR 来生成的,以评估分类器的性能。
Detection Error Trade-off (DET) Curve
检测误差权衡 (DET) 曲线
A DET curve is plotted using FRR and FAR for varying decision thresholds. To determine the region of error rates, both axes are scaled non-linearly [47]. Deviation- or logarithmic scales are the most commonly used scales in such graphs.
使用 FRR 和 FAR 针对不同的决策阈值绘制 DET 曲线。为了确定错误率区域,两个轴都被非线性缩放[47]。偏差或对数刻度是此类图中最常用的刻度。

3. Behavioural biometric modalities’ attributes and features
3. 行为生物识别模式的属性和特征

This section presents the attributes and features of behavioural biometric modalities that can be exploited for conceptualizing and designing human-to-things recognition schemes. In particular we examine behavioural biometric modalities based on HCI and natural habits that can be collected with no explicit user input using users’ smart devices, e.g., smart devices, smartwatches, etc., or external IoT sensors/nodes, e.g., pressure sensors, camera, etc.

3.1. Touch-strokes dynamics

Touch-strokes can be described as touch sequences registered by a touchscreen sensor while users navigate on touchscreen-based smart devices using their fingers [49]. Studies have shown that human musculoskeletal structure can produce finger movements that can differ from person to person [50]. Thus, a unique digital signature can be obtained from individuals’ touch-points or keystrokes collected using built-in touch sensors available in smart devices. Commonly, touch-stroke features can be categorized as spatial, timing, and motion features [51].

3.1.1. Spatial features 3.1.1.空间特征

Spatial features for touch-stroke involves physical interactions between a user fingertip and a device touchscreen surface that can be acquired when a touch event is triggered. Subsequently, a cumulative distance, i.e., a sum of lengths computed from all the consecutive touchpoints in the 2-D space, and speed, i.e., cumulative distance divided by total touch-time, can be derived from touch events [52]. Commonly used spatial features are touch positions, time-stamp, touch size, and pressure [53], [54].

3.1.2. Timing features 3.1.2.计时功能

The touch-stroke timing features generation method can utilize dwell (press or hold) and flight (latency) time. Dwell time can be defined as the time duration of a touch-event of the same key and flight time can be defined as the time interval between the touch events of two successive keys. These features are directly proportional to the number of touches on the touch-screen. As an example, Fig. 5 illustrates 30 features containing 8-Type0 dwell time features and 22-Type1 to Type4 flight time features that can be extracted from the 8 touch-sequence [55].
触摸笔画时序特征生成方法可以利用停留(按下或保持)和飞行(延迟)时间。停留时间可以定义为同一按键的触摸事件的持续时间,飞行时间可以定义为两个连续按键的触摸事件之间的时间间隔。这些功能与触摸屏上的触摸次数成正比。作为一个例子,图 5 显示了 30 个特征,其中包含 8 个类型 0 的停留时间特征和 22 个类型 1 到类型 4 的飞行时间特征,这些特征可以从 8 个触摸序列中提取[55]。

Fig. 5
  1. Download : Download high-res image (506KB)
    下载:下载高分辨率图像 (506KB)
  2. Download : Download full-size image

Fig. 5. Commonly used duration based touch-strokes timing features.
图 5. 常用的基于触摸笔画计时功能的持续时间。

The touch-stroke timing features generation method can also utilize different key-touch duration as illustrated in Fig. 6. The shortest feature-length can be termed as uni-graph, which is the timing feature extracted by taking the touch event timestamp values of the same key [56]. The timing features extracted from two, three, or more keys are termed as di-graph, tri-graph, and n-graph, respectively.
触摸笔画时序特征生成方法还可以利用不同的按键触摸持续时间,如图 6 所示。最短的特征长度可以称为单图,它是通过取触摸事件时间戳值提取的时序特征相同的键[56]。从两个、三个或更多键提取的时序特征分别称为二图、三图和n图。

Fig. 6
  1. Download : Download high-res image (132KB)
    下载:下载高分辨率图像 (132KB)
  2. Download : Download full-size image

Fig. 6. Graph based touch-strokes timing features.
图 6. 基于图形的触摸笔画计时特征。

3.1.3. Motion features 3.1.3.运动特性

Motion features can be acquired using motion sensors, such as Accelerometer, Gyroscope, Magnetometer, or gravity sensors that are available in most smart devices. Each touch event normally inflicts some movements or rotations that can be registered to generate a unique user authentication signature [57]. However, these motion features can be associated better for other user behaviors like hold- and pick-up movement [58].

3.2. Swipe 3.2.滑动

Swipe can be defined as a finite touch-events sequence that occurred as a result of users touching a smart device’s touchscreen with their finger. Smart devices provide APIs to get touch coordinates, velocity, and pressure data for each touch-point [59].
滑动可以定义为用户用手指触摸智能设备触摸屏时发生的有限触摸事件序列。智能设备提供 API 来获取每个触摸点的触摸坐标、速度和压力数据 [59]。

Some of the spatial features that can be extracted from a swipe action are the touch-points timestamp, x- and y-coordinates, velocity, and acceleration. Acceleration for each touch-point can be computed mathematically, from velocity data. The touch pressure of each touch-point determines how hard the finger was pressed on the screen, and what was the touch size. Also, trajectory length, duration, average velocity, average touch-size, start and end touch coordinates can be derived from a swipe data [60], [61]. Additionally, statistical features, such as min, max, average, standard deviation, variance, kurtosis, and skewness can be computed from each 2-D touch sequence, i.e., position, velocity, acceleration, and pressure, acquired for a swipe action [62].
可以从滑动动作中提取的一些空间特征包括触摸点时间戳、x 和 y 坐标、速度和加速度。每个接触点的加速度可以根据速度数据进行数学计算。每个触摸点的触摸压力决定了手指在屏幕上按压的力度以及触摸的大小。此外,轨迹长度、持续时间、平均速度、平均触摸大小、开始和结束触摸坐标可以从滑动数据导出[60]、[61]。此外,还可以根据针对滑动动作获取的每个二维触摸序列(即位置、速度、加速度和压力)来计算统计特征,例如最小值、最大值、平均值、标准差、方差、峰度和偏度。 62]。

3.3. Touch signature 3.3.触摸签名

Touch signature, i.e., a person signing on smart devices’ touchscreen using their finger or stylus, is similar to a handwritten signature. Although, a touch signature can utilize the features that are extracted for a swipe gesture to generate a unique identification for users specified in Section 3.2.
触摸签名,即一个人使用手指或手写笔在智能设备的触摸屏上签名,类似于手写签名。尽管如此,触摸签名可以利用为滑动手势提取的特征来为第 3.2 节中指定的用户生成唯一标识。

Typically, touch signature features can be classified as global and local features [63]. Global features include total writing time, number of strokes, and signature size. Local features include local velocity, stroke angles, etc., computed at an instance of time or for a short duration. Some of the statistical features that can be extracted for touch signature are minimum, maximum, and mean of speed, acceleration, pressure, and size of the continuous strokes [64]. Further, for each stroke in a touch signature, touch-duration, segment direction, log curvature radius, stroke length to width ratio can be extracted [65], [66].

Touch-duration can be utilized for finding similarity between touch signatures of a person. The difference between the two touch-duration sequences (Tdifference) can be computed using Tdifference=n=1N|Ts(n)Tr(n)|. Ts(n) and Tr(n) are touch-duration of nth touch sequence, respectively that are obtained from two touch signatures of a person. The direction (θi) of i-th segment having coordinates (xi,yi; xi+1,yi+1) can be calculated as θi=arctan(yi+1yixi+1xi)i=1toN. After decomposing the signature into multiple strokes, Lognormal velocity distribution vi(t) of ith stroke for a given starting time (t0i), stroke-length (Di), logtime delay (μi) and logresponse time (σi) can be obtained using Equation, i.e., |vi(t)|=Di2πσi(tt0i)exp((ln(tt0i)μi)22σi2).
触摸持续时间可用于查找人的触摸签名之间的相似性。两个触摸持续时间序列之间的差异 ( Tdifference ) 可以使用 Tdifference=n=1N|Ts(n)Tr(n)| 计算。 Ts(n) Tr(n) nth 触摸序列的触摸持续时间,分别是从一个人的两个触摸签名获得的。坐标为 ( xi,yi ; xi+1,yi+1 ) > )可以计算为 θi=arctan(yi+1yixi+1xi)i=1toN 。将签名分解为多个笔划后,给定开始时间( ith 笔划的对数正态速度分布 vi(t) t0i ),行程长度( Di ),对数时间延迟( μi )和对数响应时间( σi )可以使用方程获得,即 |vi(t)|=Di2πσi(tt0i)exp((ln(tt0i)μi)22σi2)

3.4. Hand movements 3.4.手部动作

Hand movements can be defined as a finite trajectory in 3-D space for gestures like hold, upward, downward, or snap while users perform a particular activity using their smart devices. For a user’s hand-movement action, unique user-identification-signature can be generated from collected X, Y, Z, and M coordinates. In this process, X, Y, and Z streams can be collected using sensors such as Accelerometer, Gyroscope, Magnetometer, or Gravity sensors, available in smart devices. Whereas, magnitude (M) stream can be derived mathematically, from each sensor sample (X, Y, Z), i.e.,  M=(X2+Y2+Z2).
当用户使用智能设备执行特定活动时,手部运动可以定义为 3D 空间中的有限轨迹,用于保持、向上、向下或捕捉等手势。对于用户的手势动作,可以从收集的 X Y 、 < b4> Z M 坐标。在此过程中, X Y Z 流可以使用智能设备中提供的加速度计、陀螺仪、磁力计或重力传感器等传感器来收集。然而,幅度( M )流可以通过数学方式从每个传感器样本( X Z ),即 M=(X2+Y2+Z2)

Univariate statistical features can then be extracted from each raw stream that aid to reduce the dimensionality of raw data and improve the signal-to-noise ratio [41]. Some of the statistical features, such as min (minimum value), max (maximum value), mean (average value), standard deviation (variation from the mean value), skewness (measure of the distortion or asymmetry), kurtosis (measure of the tailedness), etc., for a dataset (S) containing N values can be computed using Eqs. 1.
然后可以从每个原始流中提取单变量统计特征,这有助于降低原始数据的维数并提高信噪比[41]。一些统计特征,例如 min(最小值)、max(最大值)、mean(平均值)、标准差(与平均值的偏差)、偏度(扭曲或不对称的度量)、峰度(扭曲或不对称的度量)对于包含 N 值的数据集 ( S ) 等,可以使用等式计算: 1.

3.5. Voice 3.5.嗓音

Speech processing can be a challenging task as people have different accents, pronunciations, styles, word rates, speed of speech, speech emphasis, accent, and emotional states. Typically, a voice-based authentication system can be either text-dependent or text-independent. Fig. 7 illustrates speech processing methods encompassing speaker identification, speaker detection, and speaker verification [67].
语音处理可能是一项具有挑战性的任务,因为人们有不同的口音、发音、风格、语速、语速、语音重点、口音和情绪状态。通常,基于语音的认证系统可以是文本相关的,也可以是文本无关的。图 7 说明了包含说话人识别、说话人检测和说话人验证的语音处理方法 [67]。

Fig. 7
  1. Download : Download high-res image (381KB)
    下载:下载高分辨率图像 (381KB)
  2. Download : Download full-size image

Fig. 7. An overview of speech processing [67].
图 7. 语音处理概述 [67]。

Voice biometrics exploit human speech parametrization or pattern matching/scoring methods to generate a unique identification signature. Human speech generation involves the lungs, vocal cords, and vocal tracts [68]. When a person speaks, the air expels from the lungs passing through the vocal cords that dilate or expand allowing the airflow to produce unvoiced or voiced sound. Subsequently, the air is resonated and reshaped by the vocal tract that consists of multiple organs such as the throat, mouth, nose, tongue, teeth, and lips. The vocal cord’s modulation, interaction, and movement of these organs can alter sound waves and produce unique sounds for each person. For a sound, the phoneme is known as the smallest distinctive unit sound of a speech [69] and pitch can be referred to as a fundamental frequency [70]. Each phoneme sound can be explained as airwaves produced by the lungs that are modulated by the vocal cords and vocal tract system.

Speech parametrization transforms a speech signal into a set of feature vectors, such as Mel Frequency Cepstral Coefficients (MFCCs), mean Hilbert envelope coefficients (MHEC) [71], Power Normalized Cepstral Coefficients (PNCCs) [72], and non-negative matrix factorisation (NMF) [73]. MFCCs are widely used parametric features for automatic speech and speaker recognition systems [74]. A Mel is a unit of pitch [75]. The sound pairs that are perceptually equidistant in pitch are separated by an equal number of Mels. The mapping between frequency in Hertz and the Mel scale is linear below 1000 Hz and logarithmic above 1000 Hz. The Mel frequency (mel(f)=1127ln(1+f700)) can be computed from the raw acoustic frequency.
语音参数化将语音信号转换为一组特征向量,例如梅尔频率倒谱系数(MFCC)、平均希尔伯特包络系数(MHEC)[71]、功率归一化倒谱系数(PNCC)[72]和非负矩阵因式分解(NMF)[73]。 MFCC 是自动语音和说话人识别系统广泛使用的参数特征[74]。 Mel 是音高单位[75]。感知上音高等距的声音对被相同数量的梅尔分开。以赫兹为单位的频率与梅尔标度之间的映射在 1000 Hz 以下为线性,在 1000 Hz 以上为对数。梅尔频率 ( mel(f)=1127ln(1+f700) ) 可以根据原始声频计算出来。

To extract MFCCs, first the voice signal is pre-emphasized using a first-order high-pass filter to boost the high frequencies energy. The next step involves windowing that can be performed using the Hamming function to extract spectral features from a small window of speech. Afterward, Fast Fourier Transform (FFT) is applied to extract spectral information from the windowed signal to determine the amount of energy at each frequency band. For computing MFCCs, filter banks are created with 10 filters spaced linearly below 1000 Hz, and the remaining filters spread logarithmically, above 1000 Hz collecting energy from each frequency band. After taking the log of each of the mel spectrum values. Finally, Inverse Fast Fourier Transform (IFFT) is applied extracting the energy and 12 cepstral coefficients for each frame.
为了提取 MFCC,首先使用一阶高通滤波器对语音信号进行预加重,以增强高频能量。下一步涉及可以使用汉明函数执行的加窗,以从小语音窗口中提取频谱特征。然后,应用快速傅里叶变换 (FFT) 从加窗信号中提取频谱信息,以确定每个频带的能量量。为了计算 MFCC,滤波器组由 1000 Hz 以下线性间隔的 10 个滤波器创建,其余滤波器以对数方式扩展,在 1000 Hz 以上收集来自每个频段的能量。取每个梅尔谱值的对数后。最后,应用快速傅里叶逆变换 (IFFT) 提取每帧的能量和 12 个倒谱系数。

Pattern matching/scoring methods involves probabilistic modeling (e.g., Gaussian Mixture Model (GMM) [76], Hidden Markov Models (HMMs) [77], Joint factor analysis (JFA), i-vectors [76]), template matching (e.g., vector quantization, nearest neighbor) and deep neural network trained on various combinations of i-vectors, x-vector, feature-space maximum likelihood linear regression (fMLLR) transformation [76] or Gabor filter (GF) [78]. I-vectors are low-dimensional fixed-length speaker-and-channel dependent space that is a result of joint factor analysis [79]. For extremely short utterances, i-vectors based approaches can provide an effective speaker identification solution using different scoring methods like cosine distance or probabilistic linear discriminant analysis (PLDA). In an x-vector system, DNN is trained to extract the speaker’s voice features, and the extracted speaker embedding is called x-vector [80].
模式匹配/评分方法涉及概率建模(例如高斯混合模型(GMM)[76]、隐马尔可夫模型(HMM)[77]、联合因子分析(JFA)、i-向量[76])、模板匹配(例如、向量量化、最近邻)和深度神经网络,在 i 向量、x 向量、特征空间最大似然线性回归 (fMLLR) 变换 [76] 或 Gabor 滤波器 (GF) [78] 的各种组合上进行训练。 I 向量是低维固定长度的说话者和通道相关空间,是联合因子分析的结果[79]。对于极短的话语,基于 i 向量的方法可以使用不同的评分方法(例如余弦距离或概率线性判别分析 (PLDA))提供有效的说话人识别解决方案。在x向量系统中,训练DNN来提取说话人的语音特征,提取的说话人嵌入称为x向量[80]。

3.6. Gait 3.6.步态

Human gait is the defined as the manner and style of walking [81]. Gait can be characterized by its cadence that is measured as the number of steps per time unit. Typically, a person’s gait varies during different activities, e.g., walking, running, hopping, ascending, or descending, etc. [82]. A gait cycle, illustrated in Fig. 8, consists of two primary phases: stance and swing [83]. The stance phase is the time-period during which feet are on the ground, constitutes approximately 60% of the gait cycle. The swing phase is the time-period during which the foot is in the air, constitutes the remaining 40% of the gait cycle. A stance phase can be further divided into 1) initial-contact and loading-response, 2) mid-contact and terminal-response, and, 3) Pre-swing. Similarly, a swing phase can be divided into 1) initial, 2) mid, and 3) terminal swing [84]. Using these parameters, both time-based and spatial features can be extracted as indicated in Table 4.
人类步态被定义为行走的方式和风格[81]。步态可以通过节奏来表征,节奏以每个时间单位的步数来衡量。通常,一个人的步态在不同的活动中会有所不同,例如步行、跑步、跳跃、上升或下降等[82]。如图 8 所示,步态周期由两个主要阶段组成:站立和摆动 [83]。站立阶段是脚接触地面的时间段,约占步态周期的 60%。摆动阶段是脚在空中的时间段,构成步态周期的剩余 40%。站立阶段可进一步分为 1) 初始接触和负载响应,2) 中间接触和最终响应,以及 3) 预摆动。类似地,摆动阶段可以分为 1)初始,2)中期和 3)末端摆动 [84]。使用这些参数,可以提取基于时间和空间的特征,如表 4 所示。

Fig. 8
  1. Download : Download high-res image (368KB)
    下载:下载高分辨率图像 (368KB)
  2. Download : Download full-size image

Fig. 8. An illustration of a gait cycle.
图 8.步态周期的图示。

Table 4. Gait features. 表 4.步态特征。

#Spatial 空间Time 时间
1.Stride length (cm) 步幅(厘米)Duration of step (milli sec)
2.Step length (cm) 步长(厘米)Stride duration (milli sec)
3.Stride width or base of support (cm)
Stance phase (milli sec) 站立阶段(毫秒)
4.Internal/External Angle (deg)
内/外角 ( deg )
Swing phase (milli sec) 摆动相位(毫秒)
5.Speed (m/s or cm/s) 速度(米/秒或厘米/秒)Cadence(steps/min) 步频(步/分钟)
6.Walk ratio (cm/step/min) 步行比(厘米/步/分钟)-

Some more gait features [40] that can be analyzed for user recognition are gait variability and angular kinematics. Gait Variability (GV) can be defined as changes in gait parameters from one stride to the next. In a gait cycle, the coefficient of variation (CV) that is a measure of total variability can be calculated as root mean square (RMS) of standard deviation (σ) of the moment over stride period t mean of the absolute moment of force over stride period using Equation, i.e., CV=1ni=1nσ21ni=1n|Xi|.
可以分析用户识别的更多步态特征 [40] 包括步态变异性和角度运动学。步态变异性 (GV) 可以定义为步态参数从一步到下一步的变化。在一个步态周期中,作为总变异性度量的变异系数 (CV) 可以计算为标准差 ( σ ) 的均方根 (RMS)步幅周期内的力矩 t 使用方程计算步幅周期内绝对力力矩的平均值,即 CV=1ni=1nσ21ni=1n|Xi|

Angular Kinematics of joint angles refers to the kinematics analysis of angular motion [40]. Angular displacement (the difference between the initial and final angular position), angular velocity (change in angular position over a period of time), and angular acceleration (change in angular velocity over a period of time) can be obtained using Eqs. 2.
关节角度的角运动学是指角运动的运动学分析[40]。角位移(初始角位置与最终角位置之差)、角速度(一段时间内角位置的变化)和角加速度(一段时间内角速度的变化)可以使用等式获得。 2.

3.7. Footstep 3.7.脚步

A footstep is defined as a combination of a single left and right stride of a person. Footstep features include stride length, stride direction, timing information, acoustic and psycho-acoustic parameters, spatial positions, and relative pressure values in foot regions. These features can be captured using a range of sensors including floor-based sensors[85], such as piezoelectric sensors, switch sensors, or fabric-based pressure mapping sensors.

Ground Reaction Force (GRF) is the common feature providing a description of a person’s footstep force acquired from pressure sensors [44]. Ground Reaction Force (GRFi) per sensor can be computed by accumulating each ith sensor pressure amplitude from time t=1 to t=Tmax using Equation, i.e., GRFi=t=1TmaxPi[t].
地面反作用力(GRF)是描述从压力传感器获取的人的脚步力的常见特征[44]。每个传感器的地面反作用力 ( GRFi ) 可以通过从时间 ith 传感器压力幅度来计算/b4> t=1 t=Tmax 使用方程,即 GRFi=t=1TmaxPi[t]

Furthermore, using Eq. 3 time-series arrays, namely, average spatial pressure (SPave), cumulative spatial pressure (SPcumulative), upper (SPupper) and lower (SPlower) contours can be generated from the pressure signals acquired from N sensors for a T time-period [86].
此外,使用方程。 3个时间序列数组,分别为平均空间压力( SPave )、累积空间压力( SPcumulative )、上限( SPupper )和下部( SPlower )轮廓可以根据从 获取的压力信号生成 N 传感器用于 T 时间段[86]。
(3)SPave[t]=i=1NPi[t]SPcumulative[t]=i=1NPi[t]+i=1NPi[t1]SPupper[t]=maxi=1NSi[t]SPlower[t]=mini=1NSi[t]where, Pi[t] is the differential pressure value from the ith sensors at the time t, and, N is the total number of sensors. Footstep analysis is applicable for numerous applications, such as predicting human action, security, and surveillance at public places [86].
其中, Pi[t] ith 传感器在 t N 是传感器总数。足迹分析适用于许多应用,例如预测人类行为、公共场所的安全和监视[86]。

4. State-of-the-art in HCI and natural habits based behavioural biometrics
4. 最先进的人机交互和基于自然习惯的行为生物识别技术

This section discusses the state-of-the-art for user recognition schemes based on HCI and natural habits-based behavioural biometrics discussed in Section 2.1. We present a systematic narrative of the recent literature developing touch-stroke dynamics, swipe gesture, touch signature, hand micro-movements, voice-prints, gait, and footstep behavioural biometrics modalities for designing user recognition schemes targeting IoT applications.
本节讨论基于 HCI 和第 2.1 节中讨论的基于自然习惯的行为生物识别技术的最先进的用户识别方案。我们系统地叙述了最近开发触摸笔画动力学、滑动手势、触摸签名、手部微动作、声纹、步态和脚步行为生物识别模式的文献,用于设计针对物联网应用的用户识别方案。

Touch-stroke dynamics: User recognition methods based on touch-stroke dynamics can readily implemented in IoT endpoints such as smartphones, tablets, smartwatches, or other devices equipped with a touchscreen. Zheng et al. [53] utilized users’ tapping behavior for user verification in a passcode-enabled smartphone. They recruited 80 subjects to explore tapping behaviors using four different factors, i.e., acceleration, pressure, size, and time. They evaluated their scheme using a one-class classifier and achieved an EER of 3.65%. Further, their experiment to quantitatively measure the effect of the mimic attack revealed that only dissimilarity scores of acceleration reduced, whereas the score ranges of the other three features spread wider. Similarly, Teh et al. [54] investigated touch dynamics biometrics by extracting a basic set of timing and spatial features known as First Order Features (FOF). They derived an extended Set of Features (SOF) from the FOF features. They used both a one-class classifier (K-Nearest Neighbor (kNN), Support Vector Data Description (SVDD)), and a binary-class classifier (kNN, State Vector Machine (SVM)) for evaluation of their scheme on a dataset having 150 subjects. Through experiments, they demonstrated a reduction in impersonation attempts to 9.9% from 100% by integrating the touch dynamics authentication method into a 4-digit PIN-based authentication method in contrast to the sole use of PIN-based authentication.
触摸笔划动态:基于触摸笔划动态的用户识别方法可以轻松地在智能手机、平板电脑、智能手表或其他配备触摸屏的设备等物联网端点中实现。郑等人。 [53]利用用户的点击行为在支持密码的智能手机中进行用户验证。他们招募了 80 名受试者,利用四种不同的因素(即加速度、压力、大小和时间)来探索敲击行为。他们使用一类分类器评估了他们的方案,并取得了 3.65% 的 EER。此外,他们定量测量模仿攻击效果的实验表明,只有加速度的相异性分数降低了,而其他三个特征的分数范围扩大了。同样,Teh 等人。 [54]通过提取一组基本的时间和空间特征(称为一阶特征(FOF))来研究触摸动态生物识别。他们从 FOF 特征中派生出扩展特征集 (SOF)。他们使用一类分类器(K 最近邻 (kNN)、支持向量数据描述 (SVDD))和二元类分类器(kNN、状态向量机 (SVM))来评估数据集上的方案有150个科目。通过实验,他们证明,通过将触摸动态身份验证方法集成到基于 4 位 PIN 的身份验证方法中,与单独使用基于 PIN 的身份验证相比,可以将假冒尝试从 100% 减少到 9.9%。

Draw-a-pin is a PIN content analyzer and drawing behavior analyzer to verify the two factors of a log-in attempt [87]. The system extracts touch information, such as x-coordinates, y-coordinates, finger pressure, and touch area size, from each 4-digit pin. They claim the scheme is resilient against shoulder surfing attacks and achieved an EER of 4.84% using the Dynamic Time Warping (DTW) algorithm on 20 subjects. Similar to the draw-a-pin approach, Tolosana et al. [88] suggested replacing conventional authentication systems based on PIN and One-Time Passwords (OTP) with a scheme that allows users to draw each digit of the password on the device’s touchscreen. They created an e-BioDigit database consisting of 93 subjects to conduct their experiment. The authors evaluated the scheme using DTW by combining with the Sequential Forward Feature Selection (SFFS) function selection algorithm and Recurrent Neural Networks (RNNs) deep learning technology that exploited various touch features; they achieved an EER of 4%.
Draw-a-pin 是一个 PIN 内容分析器和绘图行为分析器,用于验证登录尝试的两个因素 [87]。系统从每个 4 位引脚中提取触摸信息,例如 x 坐标、y 坐标、手指压力和触摸区域大小。他们声称该方案能够抵御肩窥攻击,并在 20 名受试者上使用动态时间规整 (DTW) 算法实现了 4.84% 的 EER。与拔针方法类似,Tolosana 等人。 [88]建议用一种允许用户在设备触摸屏上绘制密码的每一位数字的方案来取代基于 PIN 和一次性密码 (OTP) 的传统身份验证系统。他们创建了一个包含 93 名受试者的 e-BioDigit 数据库来进行实验。作者结合顺序前向特征选择(SFFS)函数选择算法和利用各种触摸特征的循环神经网络(RNN)深度学习技术,使用DTW对该方案进行了评估;他们的 EER 达到了 4%。

Multi-touch authentication with TFST (touch with fingers straight and together) gestures is a simple and reliable authentication scheme for devices equipped with multi-touch screens [58]. The scheme exploits both hand geometry and behavioural characteristics and the authors collected a large multi-touch dataset from 161 subjects. They achieved an EER of 5.48% (5 training samples) using one-class SVM and kNN classifiers. Furthermore, they performed a security analysis for a zero-effort attack, smudge attack, shoulder surfing attack, and statistical attack. Touch-stroke dynamics is a relatively recent behavioural biometrics when compared to well established behavioural biometrics such as signature verification. Table 5 compares user recognition schemes based on touch-strokes dynamics.
对于配备多点触摸屏的设备来说,使用 TFST(手指并拢触摸)手势的多点触摸身份验证是一种简单可靠的身份验证方案[58]。该方案利用了手部几何形状和行为特征,作者从 161 名受试者中收集了大型多点触摸数据集。他们使用一类 SVM 和 kNN 分类器实现了 5.48% 的 EER(5 个训练样本)。此外,他们还对零努力攻击、涂抹攻击、肩窥攻击和统计攻击进行了安全分析。与签名验证等成熟的行为生物识别技术相比,触摸动作动力学是一种相对较新的行为生物识别技术。表 5 比较了基于触摸笔划动态的用户识别方案。

Table 5. User recognition schemes based on touch-strokes dynamics.
表 5. 基于触摸笔划动态的用户识别方案。

Study 学习Methodology/Features 方法/特点Algorithm/Classifier 算法/分类器Dataset 数据集Performance 表现
Li et al. [89], 2021
李等人。 [89],2021
Single touch, touch movement and multi-touch
SVM60 subjects 60 个科目Average error rate 2.9%
平均错误率 2.9%
Teh et al. [54], 2019
泰等人。 [54],2019
FOF and SOF FOF和SOFkNN, SVDD, and SVM kNN、SVDD 和 SVM150 subjects 150 个科目Impersonation rate = 9.9%
冒充率 = 9.9%
Zheng et al. [53], 2014
郑等人。 [53],2014
Tapping behaviors 敲击行为one-class machine learning technique
80 subjects 80 个科目EER = 3.65% 能效比=3.65%
Song at al. [58], 2017
歌曲在等。 [58],2017
Multi-touch with TFST 带有 TFST 的多点触控One-class SVM and kNN 一类 SVM 和 kNN161 subjects 161 科目EER = 5.48% (5 training samples)
EER = 5.48%(5 个训练样本)
Tolosana et al. [88], 2017
托洛萨纳等人。 [88],2017
Handwritten numerical digits using finger-touch
DTW combined with the SFFS and RNNs
DTW 与 SFFS 和 RNN 相结合
e-BioDigit [90] (93 subjects)
e-BioDigit [90](93 个科目)
EER = 4% 能效比=4%

Swipe gesture: A swipe gesture (collection of touch-strokes from a touch-down to touch-release) can be processed for user recognition. SwipeVlock authenticates users based on their way of swiping the phone screen with a background image [61]. The scheme was evaluated using a decision tree, Naive Bayes (NB), SVM, and Back Propagation Neural Network (BPNN) on 150 subjects and achieved a success rate of 98%. DriverAuth collected and encoded a sequence of touch-events when a user swipes on the touchscreen using their finger. It achieved a TAR of 87% using Quadratic SVM (Q-SVM) on a dataset of 86 subjects. Jain et al. [57] analyzed swipe gestures, such as left-to-right swipe (L2R), right-to-left swipe (R2L), scroll up (SU), scroll down (SD), zoom in (ZI), zoom out (ZO) and single tap (ST), subsequently, extracting xy coordinates, accelerometer, orientation sensor readings, and area covered by a finger to design an authentication scheme. The scheme recruited 104 subjects for evaluation and 30 subjects for performance verification. Using a modified Hausdorff distance (MHD), they achieved an EER of 0.31% for combined gestures using score level fusion.
滑动手势:可以处理滑动手势(从触摸到触摸释放的触摸笔划的集合)以供用户识别。 SwipeVlock 根据用户使用背景图像滑动手机屏幕的方式对用户进行身份验证 [61]。该方案使用决策树、朴素贝叶斯 (NB)、SVM 和反向传播神经网络 (BPNN) 对 150 名受试者进行了评估,成功率达到 98%。当用户用手指在触摸屏上滑动时,DriverAuth 收集并编码一系列触摸事件。它在 86 名受试者的数据集上使用二次 SVM (Q-SVM) 实现了 87% 的 TAR。贾恩等人。 [57]分析了滑动手势,例如从左向右滑动(L2R)、从右向左滑动(R2L)、向上滚动(SU)、向下滚动(SD)、放大(ZI)、缩小( ZO)和单击(ST),随后提取 xy 坐标、加速度计、方向传感器读数和手指覆盖的区域来设计身份验证方案。该计划招募了 104 名受试者进行评估,30 名受试者进行绩效验证。使用修改后的豪斯多夫距离 (MHD),他们使用分数级别融合实现了组合手势的 EER 为 0.31%。

Ellavarason et al. [60] proposed a swipe gesture authentication and collected a dataset under four scenarios, i.e., sitting (room and bus) and walking (outdoor and treadmill). They used SVM, kNN, and NB are used to evaluate the robustness of swipe gestures and achieved an ERR of 1% (sitting in a room), 30% (sitting in a bus), 23% (walking on a treadmill), 27% (walking outdoor) on 50 subjects. According to Poze et al. [91], horizontal strokes hold more user-specific information and are more discriminating than vertical strokes. They investigated a statistical approach based on adapted Gaussian Mixture Models (GMM) for swipe gestures and achieved an EER of 20% (40 training samples) using a dataset with 90 subjects. Garbuz et al. [92] proposed an approach that analyzed both swipes and taps to provide continuous authentication. The one-class classification model is generated using one-class SVM. The scheme can detect an impostor in 2–3 gestures, whereas the legitimate user is blocked on average after 115–116 gestures.
埃拉瓦拉森等人。 [60]提出了一种滑动手势认证,并收集了四种场景下的数据集,即坐着(房间和公共汽车)和步行(户外和跑步机)。他们使用 SVM、kNN 和 NB 来评估滑动手势的鲁棒性,并取得了 1%(坐在房间里)、30%(坐在公共汽车上)、23%(在跑步机上行走)的 ERR,27 %(户外行走)50 个科目。根据 Poze 等人的说法。 [91],水平笔划包含更多用户特定信息,并且比垂直笔划更具辨别力。他们研究了一种基于适用于滑动手势的自适应高斯混合模型 (GMM) 的统计方法,并使用包含 90 名受试者的数据集实现了 20% 的 EER(40 个训练样本)。加尔布兹等人。 [92]提出了一种分析滑动和点击以提供连续身份验证的方法。一类分类模型是使用一类支持向量机生成的。该方案可以在 2-3 个手势中检测到冒名顶替者,而合法用户平均在 115-116 个手势后就会被阻止。

Another scheme involved the extraction of temporal information from consecutive touch-strokes [93]. For evaluation, they temporal Regression Forest (TRF) architecture and achieved an EER of 4%, 2.5% on the Serwadda and Frank datasets, having 190 and 41 subjects, respectively. Kumar et al. [94] proposed a multimodal scheme that exploited swiping gestures, typing behavior, phone movement patterns while typing/swiping, and their possible fusion at the feature- and score-level for authenticating smartphone users, continuously. A multi-template classification framework (MTCF) is implemented for evaluation. They achieved an accuracy of 93.33% and 89.31% using feature level and score level fusion, respectively on 28 subjects. Table 6 compares user recognition schemes based on swipe gesture.
另一种方案涉及从连续的触摸笔画中提取时间信息[93]。为了进行评估,他们采用了时间回归森林 (TRF) 架构,并在 Serwadda 和 Frank 数据集(分别有 190 名受试者和 41 名受试者)上实现了 4% 和 2.5% 的 EER。库马尔等人。 [94]提出了一种多模式方案,该方案利用滑动手势、打字行为、打字/滑动时的手机移动模式,以及它们在功能和分数级别上可能的融合,以连续验证智能手机用户的身份。采用多模板分类框架(MTCF)进行评估。他们在 28 名受试者上使用特征级别和分数级别融合分别获得了 93.33% 和 89.31% 的准确率。表 6 比较了基于滑动手势的用户识别方案。

Table 6. User recognition schemes based on swipe.
表 6. 基于滑动的用户识别方案。

Study 学习Methodology/Features 方法/特点Algorithm/Classifier 算法/分类器Dataset 数据集Performance 表现
Jain et al. [57], 2021
贾恩等人。 [57],2021
Touchscreen gestures (L2R, R2L, SU, SD, ZI, ZO, and ST)
触摸屏手势(L2R、R2L、SU、SD、ZI、ZO 和 ST)
Modified MHD 改良磁流体动力学104 subjects for evaluation and 30 subjects for performance verification
EER = 0.31% for combined gestures using score level fusion
使用分数级别融合的组合手势的 EER = 0.31%
Gupta et al. [59], 2019
古普塔等人。 [59],2019
Touch-events sequence 触摸事件序列Q-SVM Q支持向量机86 subjects [95] 86 个科目 [95]TAR = 87% 焦油 = 87%
Ellavarason et al. [60], 2020
埃拉瓦拉森等人。 [60],2020
Swipe gesture in four scenarios - sitting (room and bus) and walking (outdoor and treadmill)
四种场景下的滑动手势 - 坐着(房间和公共汽车)和步行(户外和跑步机)
SVM, kNN, and NB SVM、kNN 和 NB50 subjects 50 个科目ERR = 1% (sitting in room), 30%(sitting in bus), 23% (walking on treadmill), 27% (walking outdoor)
ERR = 1%(坐在房间里)、30%(坐在公交车上)、23%(在跑步机上行走)、27%(在室外行走)
Li et al. [61], 2020
李等人。 [61],2020
Swipe on an image 在图像上滑动Decision tree, NB, SVM, and BPNN
150 subjects 150 个科目Success Rate = 98% 成功率 = 98%
Pozo et al. [91]. 2017
波佐等人。 [91]。 2017年
Horizontal and vertical strokes
GMM190 subjects 190 科目EER = 20% (40 training samples)
EER = 20%(40 个训练样本)
Kumar et al. [94], 2016
库马尔等人。 [94],2016
Swipe, typing behavior, phone movement patterns
MTCF28 subjects 28 个科目Accuracy = 93.33% (feature level fusion), 89.31% (score level fusion)
准确率 = 93.33%(特征级融合)、89.31%(分数级融合)
Ooi et al. [93], 2019
奥伊等人。 [93],2019
Touch-strokes temporal information
TRFSerwadda (190 subjects), Frank [96] (41 subjects)
Serwadda (190 科目), Frank [96] (41 科目)
EER = 4%, 2.5% 能效比 = 4%, 2.5%

Touch-signature: Touch-signature using a finger or stylus on a touchscreen device is emerging as an alternative to an all-time acceptable handwritten signature for user recognition. Features explained in Section 3.3 can be exploited to identify a user for a number of security-sensitive applications, such as hotel bookings, online-banking, and shopping thereby helping minimize fraudulent activities.
触摸签名:在触摸屏设备上使用手指或手写笔进行触摸签名正在成为用户识别的一直可接受的手写签名的替代方案。可以利用第 3.3 节中解释的功能来识别许多安全敏感应用程序的用户身份,例如酒店预订、在线银行和购物,从而有助于最大限度地减少欺诈活动。

Tolosana et al. [65] proposed an on-line signature verification system that is adaptable to the signature complexity level. In their proposed approach, a signature complexity detector based on the number of lognormals from the Sigma LogNormal writing generation model, and a time function extraction module are generated for each complexity level. Then, the DTW algorithm is used to compute the similarity between the time functions from the input signature and training signatures of the claimed user. The scheme achieved an EER of 2.5% and 5.6% on BiosecurID (pen scenario of 400 subjects) and BioSign (pen and finger scenario of 65 subjects) datasets, respectively. Yoshida et al. [66] analyzed touch-strokes duration and segments’ directions of signatures using two Japanese characters. An objective measure of the difference between two sequences of touching duration is used to evaluate the similarity and the scheme achieved an EER of 7.1% using 10 subjects. Gomez et al. [97] proposed to improve the performance of online signature verification systems based on the Kinematic Theory of rapid human movements and its associated Sigma LogNormal model. The authors used the BiosecurID multimodal database of 400 subjects having 6400 genuine signatures and 4800 skilled forgeries for the evaluation of their schemes using DTW.
托洛萨纳等人。 [65]提出了一种适应签名复杂程度的在线签名验证系统。在他们提出的方法中,为每个复杂性级别生成基于 Sigma LogNormal 写入生成模型的对数正态数的签名复杂性检测器和时间函数提取模块。然后,使用 DTW 算法计算输入签名和所声明用户的训练签名的时间函数之间的相似度。该方案在 BiosecurID(400 名受试者的笔场景)和 BioSign(65 名受试者的笔和手指场景)数据集上分别实现了 2.5% 和 5.6% 的 EER。吉田等人。 [66]使用两个日语字符分析了签名的触摸笔画持续时间和片段方向。使用对两个触摸持续时间序列之间差异的客观测量来评估相似性,该方案使用 10 名受试者实现了 7.1% 的 EER。戈麦斯等人。 [97]提出基于人体快速运动的运动学理论及其相关的 Sigma LogNormal 模型来提高在线签名验证系统的性能。作者使用包含 400 名受试者的 BiosecurID 多模态数据库(其中有 6400 个真实签名和 4800 个熟练的伪造签名)来使用 DTW 评估他们的方案。

Ren et al. [98] proposed a signature verification system leveraging a multi-touch screen for mobile transactions by extracting critical segments to capture a user’s intrinsic signing behavior for accurate signature verification. They applied DTW to calculate an optimal match between two temporal sequences with different lengths, and then measure the similarity between them. On 25 subjects, an EER of 2%, 1%, and 3% for single-finger, two-finger, and under the observation and imitation attack scenarios, respectively achieved. Al-Jarrah et al. [99] proposed anomaly detectors, such as STD Z-Score Anomaly Detector, Average Absolute Deviation (AAD) Anomaly Detector, and Median Absolute Deviation (MAD) Anomaly Detector, for signature verification. Using distance functions for evaluation, they achieved an EER between 3.21% to 5.44% for skilled forgeries and 4.74% to 6.31% for random forgeries among 55 subjects. Behera et al. [100] proposed an approach based on spot signature within a continuous air writing captured through Leap motion depth sensors. The processed signatures are represented using convex hull vertices and DTW is selected for performance verification of the spotted signatures. The authors achieved an accuracy of 80% on 20 subjects. Ramachandra et al. [101] proposed user verification using a smartwatch-based writing pattern or style that exploited accelerometer data acquired from 30 participants. The accelerometer data is further transformed using 2D Continuous Wavelet Transform (CWT) and deep features extracted using the pre-trained ResNet50. Table 7 compares user recognition schemes based on touch signature.
任等人。 [98]提出了一种签名验证系统,利用多点触摸屏进行移动交易,通过提取关键片段来捕获用户的内在签名行为,以实现准确的签名验证。他们应用 DTW 来计算两个不同长度的时间序列之间的最佳匹配,然后测量它们之间的相似度。在 25 名受试者中,单指、两指、观察和模仿攻击场景的 EER 分别达到 2%、1% 和 3%。阿尔-贾拉等人。 [99]提出了用于签名验证的异常检测器,例如 STD Z 得分异常检测器、平均绝对偏差(AAD)异常检测器和中值绝对偏差(MAD)异常检测器。使用距离函数进行评估,他们在 55 名受试者中实现了熟练伪造的 EER 在 3.21% 至 5.44% 之间,而随机伪造的 EER 在 4.74% 至 6.31% 之间。贝赫拉等人。 [100]提出了一种基于通过 Leap 运动深度传感器捕获的连续空中书写中的点签名的方法。处理后的签名使用凸包顶点表示,并选择 DTW 来验证发现的签名的性能。作者对 20 名受试者的准确率达到了 80%。拉马钱德拉等人。 [101]建议使用基于智能手表的书写模式或风格进行用户验证,该模式或风格利用从 30 个参与者获取的加速度计数据。使用 2D 连续小波变换 (CWT) 进一步转换加速度计数据,并使用预训练的 ResNet50 提取深度特征。表 7 比较了基于触摸签名的用户识别方案。

Table 7. User recognition schemes based on touch signature.
表 7. 基于触摸签名的用户识别方案。

Study 学习Methodology/Features 方法/特点Algorithm/Classifier 算法/分类器Dataset 数据集Performance 表现
Tolosana et al. [65], 2020
托洛萨纳等人。 [65],2020
Time functions for different complexity, Lognormals from Sigma LogNormal
不同复杂度的时间函数,来自 Sigma LogNormal 的对数正态
DTWBiosecurID (pen scenario of 400 subjects), BioSign (pen and finger scenario of 65 subjects)
BiosecurID(400 名受试者的笔场景)、BioSign(65 名受试者的笔和手指场景)
EER = 2.5%, 5.6% 能效比=2.5%、5.6%
Al et al. [99], 2019
艾尔等人。 [99],2019
finger-drawn signature 手指画签名Distance-based functions 基于距离的功能55 subjects 55 个科目EER = 3.21% to 5.44% (Skilled Forgery), 4.74% to 6.31% (Random Forgery)
EER = 3.21% 至 5.44%(熟练伪造),4.74% 至 6.31%(随机伪造)
Van et al. [87], 2017
范等人。 [87],2017
Touch information from 4-digit pin drawing
来自 4 位引脚图的触摸信息
DTW20 subjects