2020-12-30: UI Automation: A walkthrough with UI Automation 2020-12-30：UI 自动化：UI 自动化演练

Fig: a demonstration of UI Automation in Notepad
图：记事本中UI自动化的演示

Essential criteria for accessibility assistance in the application are programmatic access and keyboard access. To test accessibility for people with different disabilities and limitations or those who prefer to use a keyboard, it is important that you test the accessibility of your Windows applications, assistive technology (AT) tools, and user interface (UI) frameworks. You will not be able to use your application for users with visual, learning, dexterity/mobility, and with language/commune impairments or disabilities, without appropriate access through AT such as screen readers and on-screen keyboards.
应用程序中辅助功能帮助的基本标准是编程访问和键盘访问。若要测试具有不同残障和限制的人或喜欢使用键盘的人的辅助功能，请务必测试 Windows 应用程序、辅助技术 （AT） 工具和用户界面 （UI） 框架的辅助功能。如果没有通过 AT 进行适当的访问（如屏幕阅读器和屏幕键盘），您将无法为具有视觉、学习、灵巧/行动能力以及语言/社区障碍或残疾的用户使用您的应用程序。

In the last blog, I talked about some high-level concepts on UI Automation (UIA). In this walkthrough, I’ll give a snippet on how to use UIA on Notepad. You can write your script using C++, C#, or Python. Microsoft uses C++ and C# for their UI Automation and all kinds of testing. However, most of the developers use Python, since it is convenient to use as well as it can easily pipeline with another machine learning-based work. I’ll use two UIA based Python libraries for this demonstration: 
在上一篇博客中，我谈到了有关 UI 自动化 （UIA） 的一些高级概念。在本演练中，我将介绍如何在记事本上使用 UIA。可以使用 C++、C# 或 Python 编写脚本。Microsoft 使用 C++ 和 C# 进行 UI 自动化和各种测试。然而，大多数开发人员都使用 Python，因为它使用起来很方便，并且可以轻松地与其他基于机器学习的工作进行管道连接。我将使用两个基于 UIA 的 Python 库进行此演示：

• uiautomation UI自动化
• Pywinauto 皮温汽车

Uiautomation library supports UI automation for the applications which implemented UI automation provider, such as MFC, Windows Form, WPF, Modern UI(Metro UI), Qt, and Firefox. If you run it for the first time on Python, you need to run the below command first:
Uiautomation 库支持实现 UI 自动化提供程序的应用程序的 UI 自动化，例如 MFC、Windows 窗体、WPF、现代 UI（Metro UI）、Qt 和 Firefox。如果是第一次在 Python 上运行，需要先运行以下命令：

pip install uiautomation

view raw uiautomation hosted with ❤ by GitHub

Notepad is a generic Microsoft Windows text editor that allows you to create, open, and read files with plaintext. I will automate this Notepad with UIA by using the uiautomation library. My plan is to write a Python script to select and customize font, write a sentence, check with each keyboard character, and finally, save the Notepad file without using a mouse and keyboard.
记事本是一个通用的 Microsoft Windows 文本编辑器，允许您创建、打开和读取带有纯文本的文件。我将使用 uiautomation 库通过 UIA 自动执行此记事本。我的计划是编写一个 Python 脚本来选择和自定义字体，写一个句子，检查每个键盘字符，最后，在不使用鼠标和键盘的情况下保存记事本文件。

	# Initialize library
	import os
	import sys
	import time
	import subprocess
	import ctypes
	import uiautomation as auto
	# Open Notepad
	subprocess.Popen('notepad')
	# Identify Notepad Class
	window = auto.WindowControl(searchDepth = 1, ClassName = 'Notepad', RegexName = '.* - Notepad')
	#Identify the screensize
	screenWidth, screenHeight = auto.GetScreenSize()
	window.MoveWindow(screenWidth // 4, screenHeight // 4, screenWidth // 2, screenHeight // 2)
	window.SetActive()
	#Find the format menuitem
	window.MenuItemControl(Name = 'Format').Click()
	#Click the Font within format menuitem
	window.MenuItemControl(Name = 'Font...').Click()
	windowFont = window.WindowControl(Name = 'Font')
	# Select Font Family
	windowFont.ComboBoxControl(AutomationId = '1136').Select('Trebuchet MS')
	# Select Font Style
	windowFont.ComboBoxControl(AutomationId = '1137').Select('Bold Italic')
	# Select Font Size
	windowFont.ComboBoxControl(AutomationId = '1138').Select('28')
	# Confirm the selection
	windowFont.ButtonControl(Name = 'OK').Click()
	edit = auto.EditControl(searchFromControl = window) #or edit = window.EditControl()
	edit.Click(waitTime = 0)
	#Typing rich text on notepad
	edit.SendKeys('Hello from UI Automation!!')
	edit.SendKeys('{Ctrl}{A}', 0.2, 0)
	window.MenuItemControl(Name = 'Format').Click()
	window.MenuItemControl(Name = 'Font...').Click()
	windowFont = window.WindowControl(Name = 'Font')
	windowFont.ComboBoxControl(AutomationId = '1136').Select('Segoe UI Symbol')
	windowFont.ComboBoxControl(AutomationId = '1138').Select('18')
	windowFont.ButtonControl(Name = 'OK').Click()
	edit.SendKeys('{Ctrl}{End}{Enter}Lets test with keyboard{! 4}{ENTER}', 0.2, 0)
	edit.SendKeys('{Enter 2}0123456789{Enter}', waitTime = 0)
	edit.SendKeys('{Enter}ABCDEFGHIJKLMNOPQRSTUVWXYZ{Enter}', waitTime = 0)
	edit.SendKeys('{Enter}abcdefghijklmnopqrstuvwxyz{Enter}', waitTime = 0)
	edit.SendKeys('{Enter}`~!@#$%^&*()-_=+{Enter}', waitTime = 0)
	edit.SendKeys('{Enter}[]{{}{}}\\|;:\'\",<.>/?{Enter}{Ctrl}a')
	# Click close button before exit
	window.ButtonControl(searchDepth=2, Name='Close').Click()
	# Name the notepad
	window.ButtonControl(Name="Save").Click()
	auto.SendKeys('UIA')
	#Exit
	window.ButtonControl(Name="Save").Click()

view raw notepad.py hosted with ❤ by GitHub

A video demonstration of code is shown here:
代码的视频演示如下所示：

(An alternative version of the video)
（视频的替代版本）

In the video, you can see, once I clicked the compiled button on Visual Studio Code, it will open a Notepad. In the Notepad, it will search the “Format” menu item. Once it finds it, it will click and go to the “Font…” element. A dialog box will popup and within the dialog, you can see all the font-related configuration is shown for a Notepad. After selecting the recommended font configuration, it will start to write on a Notepad. Once it writes the first sentence, it will change the font configuration again to show even working on a sentence or paragraph, you can still change your font configuration. After changing it, it will test with all the character from the keyboard, finally, it will save the Notepadon the desktop and name it "UIA". This is one approach to work with the UIA library.
在视频中，你可以看到，一旦我点击了 Visual Studio Code 上的编译按钮，它就会打开一个记事本。在记事本中，它将搜索“格式”菜单项。找到它后，它将单击并转到“字体...”元素。将弹出一个对话框，在对话框中，您可以看到记事本显示所有与字体相关的配置。选择推荐的字体配置后，它将开始在记事本上书写。一旦它写了第一句话，它就会再次更改字体配置以显示，即使在处理一个句子或段落时，您仍然可以更改字体配置。更改后，它将使用键盘上的所有字符进行测试，最后，它将记事本保存在桌面上并将其命名为“UIA”。这是使用 UIA 库的一种方法。

Pywinauto, another Python library, automates the Microsoft Windows UI. It allows you to send mouse and keyboard actions to windows dialog and control type simplest way for both Windows and Linux. If you are run Pywinauto for the first time, you need to install this library by executing this command first:
Pywinauto是另一个Python库，可自动执行Microsoft Windows UI。它允许您将鼠标和键盘操作发送到 Windows 对话框，并以最简单的方式控制 Windows 和 Linux。如果你是第一次运行 Pywinauto，你需要先执行以下命令来安装这个库：

	pip install pywinauto #Python
	conda install -c conda-forge pywinauto #Anaconda

view raw pywinauto.py hosted with ❤ by GitHub

Pywinauto needs a few dependencies to support its work:
Pywinauto 需要一些依赖项来支持其工作：

1. pyWin32 pyWin32的
2. comtypes
3. six
4. Pillow 枕头

After installing dependency and the package, you are ready to write your first script. 
安装依赖项和包后，即可编写第一个脚本。

The first part of the script is to import the required module. Let's begin with the import of the Pywinauto module application class. Application for each automation process is the starting point.
脚本的第一部分是导入所需的模块。让我们从导入 Pywinauto 模块应用程序类开始。每个自动化过程的应用都是起点。

from pywinauto.application import Application

You also have to import the following code if you want to call basic user input functions like the mouse and the keyboard:
如果要调用基本用户输入函数（如鼠标和键盘），还必须导入以下代码：

import pywinauto.mouse as mouse
import pywinauto.keyboard as keyboard

The next step is to connect the application instance to automate it. There are two ways you can do it.
下一步是连接应用程序实例以自动执行它。有两种方法可以做到这一点。

notepadapp = Application(backend="uia").start("notepad.exe")

or

notepadapp = Application(backend="uia").connect(path=r"C:\Windows\system32\notepad.exe")

"start()" is used while you're not running the program and need to start it. It accepts a string that can contain the argument for the command line and a timeout. Only when your program takes too long to start is the timeout parameter needed whereas "connect()" is used while a running procedure is being attempted. To specify an already running application, you must assign one of the following: process, handle, path, or any combination of the parameters that specify a window.
“start（）”在您不运行程序并需要启动它时使用。它接受一个字符串，该字符串可以包含命令行的参数和超时。只有当程序启动时间过长时，才需要超时参数，而“connect（）”则在尝试运行过程时使用。若要指定已运行的应用程序，必须分配以下参数之一：进程、句柄、路径或指定窗口的参数的任意组合。

The next phase is to recognize all available controls by running the code below.
下一阶段是通过运行以下代码来识别所有可用的控件。

notepad_window = notepadapp.window(title='Untitled - Notepad')
notepad_window.print_control_identifiers()

It will print our all the available control type in a parent-child hierarchical tree format for each node where "Untitled - Notepad" is the root. You will automatically use various forms of controls. "Button", "Edit", and "MenuItem" are the most common buttons to use. Each control has a function call of its own. For further detail, see the documentation.
它将以父子分层树格式为每个节点打印所有可用的控件类型，其中“Untitled - Notepad”是根。您将自动使用各种形式的控件。“Button”、“Edit”和“MenuItem”是最常用的按钮。每个控件都有自己的函数调用。有关详细信息，请参阅文档。

Fig: Tree representation of Notepad 
图：记事本的树表示

Let's test with some sentences with the keyboard.
让我们用键盘测试一些句子。

dialog = notepadapp['Untitled - Notepad']
dialog.type_keys('Hello{SPACE}from{SPACE}PYWINAUTO')
dialog.type_keys('{ENTER}Lets test with keyboard')
dialog.type_keys('{ENTER}0123456789')
dialog.type_keys('{ENTER}ABCDEFGHIJKLMNOPQRSTUVWXYZ')
dialog.type_keys('{ENTER}abcdefghijklmnopqrstuvwxyz')
dialog.type_keys('{ENTER}`~!@#$%^&*()-_=+')
dialog.type_keys('{ENTER}[]{{}{}}\\|;:\'\",<.>/?')

If you execute it, you will find that "Hello from PYWINAUTO" is the final text appearing on the Notepad, where there is no place for both spaces. This is typical since the keyboard entry is based on this module. This can quickly be overcome by inserting the correct key codes in the string. For further information please visit the Pywinauto documentation.
如果你执行它，你会发现“来自PYWINAUTO的你好”是出现在记事本上的最终文本，其中两个空格都没有位置。这是典型的，因为键盘输入基于此模块。这可以通过在字符串中插入正确的键代码来快速克服。有关更多信息，请访问 Pywinauto 文档。

Similarly, you can use specific coordination to move the mouse to a particular position. Here, I will the close button of the Notepad by clicking the cross-section of the Notepad. To get a specific location, you need to run this code below:
同样，您可以使用特定的协调将鼠标移动到特定位置。在这里，我将通过单击记事本的横截面来关闭记事本的按钮。若要获取特定位置，需要运行以下代码：

notepad_window["Close"].rectangle()

The output will show the top, bottom, left, right positions of the "Close" button.
输出将显示“关闭”按钮的顶部、底部、左侧、右侧位置。

To click this, you need to focus the dialog first, then use this coordination to click the "Close" button.
要单击此按钮，您需要先聚焦对话框，然后使用此协调单击“关闭”按钮。

dialog.set_focus()
mouse.click(coords=(1608, 267, 1655, 297))

If you want to move from one menu item or browse within it, you have to use "menu_select" to proceed. For example, like the previous example of "uiautomation", if we are looking for font configuration, we need to run this code:
如果要从一个菜单项移动或在其中浏览，则必须使用“menu_select”才能继续。例如，像前面的“uiautomation”示例一样，如果我们正在寻找字体配置，我们需要运行以下代码：

fmenu = dialog.menu_select('Format->Font...')
fdlg = notepadapp.Font
ftype = fdlg.ComboBox
ftype.select('Trebuchet MS')
ftype = fdlg.ComboBox2
ftype.select('Bold Italic')
fsize = fdlg.ComboBox3
fsize.type_keys('28')
fdlg.OK.click() 

A quick recap of what we have so far. First, we install the "uiautomation" and "Pywinauto" module and demonstrate the installation processes of each module. We also examined in detail the fundamental use of the module. I used both libraries in Notepad to automate its core elements as an example in this case. We have also written some code for automating basic acts like clicking and selecting the menu. Next, we have attempted manual automation using the integrated mouse and the keyboard that offers greater versatility than normal. Thank you for reading this post, and I look forward to seeing you again!
快速回顾一下我们目前所拥有的。首先，我们安装 “uiautomation” 和 “Pywinauto” 模块，并演示每个模块的安装过程。我们还详细研究了该模块的基本用途。在本例中，我使用记事本中的两个库来自动化其核心元素作为示例。我们还编写了一些代码来自动执行基本操作，例如单击和选择菜单。接下来，我们尝试使用集成鼠标和键盘进行手动自动化，其多功能性比平时更高。感谢您阅读这篇文章，我期待再次见到您！

my GitHub repo: https://github.com/javedulferdous/UIABlog
我的 GitHub 存储库：https://github.com/javedulferdous/UIABlog

-- Md Javedul Ferdous(@jaf_ferdous)
-- Md Javedul Ferdous（@jaf_ferdous）