deep_reinforcement_learning_fo-183ae653-e302-46bb-993b-53ac63dce2ed

Abstract 摘要

Table of Contents 目录

Acknowledgements 致谢

Chapter 2 Notation, Background and Literature
第 2 章 符号、背景和文献

• 2.4 Conclusion 2.4 结论

Chapter 4 Formulating the Learning Objevtive
第 4 章 制定学习目标

* 4.1 Introduction
* 4.2 Related Work
* 4.3 Preliminaries and Notations
* 4.4 Method
	* 4.4.1 Agent Design
	* 4.4.2 Learning Process
* 4.5 Justification of RL agent
	* 4.5.1 Justification for State Design
		* 4.5.1.1 General description of traffic movement process as a Markov chain
		* 4.5.1.2 Specification with proposed state definition
	* 4.5.2 Justification for Reward Design
		* 4.5.2.1 Stabilization on traffic movements with proposed reward.
		* 4.5.2.2 Connection to throughput maximization and travel time minimization.
* 4.6 Experiment
	* 4.6.1 Dataset Description
	* 4.6.2 Experimental Settings
		* 4.6.2.1 Environmental settings
		* 4.6.2.2 Evaluation metric
		* 4.6.2.3 Compared methods
	* 4.6.3 Performance Comparison
	* 4.6.4 Study of PressLight
		* 4.6.4.1 Effects of variants of our proposed method
		* 4.6.4.2 Average travel time related to pressure.
	* 4.6.5 Performance on Mixed Scenarios
		* 4.6.5.1 Heterogeneous intersections
		* 4.6.5.2 Arterials with a different number of intersections and network
	* 4.6.6 Case Study
		* 4.6.6.1 Synthetic traffic on the uniform, uni-directional flow
			* 4.6.6.1.1 Performance comparison
			* 4.6.6.1.2 Policy learned by RL agents
		* 4.6.6.2 Real-world traffic in Jinan
* 4.7 Conclusion

* 5.1 Introduction
* 5.2 Related Work
* 5.3 Problem Definition
* 5.4 Method
	* 5.4.1 Observation Embedding
	* 5.4.2 Graph Attention Networks for Cooperation
		* 5.4.2.1 Observation Interaction
		* 5.4.2.2 Attention Distribution within Neighborhood Scope
		* 5.4.2.3 Index-free Neighborhood Cooperation
		* 5.4.2.4 Multi-head Attention
	* 5.4.3 Q-value Prediction
	* 5.4.4 Complexity Analysis

Chapter 6 Learning to Simulate
第 6 章 学习模拟

• 6.1 Introduction 6.1 导言
• 6.2 Related Work 6.2 相关工作
• 6.3 Preliminaries  6.3 前言
  • 6.4 Method  6.4 方法
    • 6.4.1 Basic GAIL Framework
      6.4.1 GAIL 基本框架
    • 6.4.2 Imitation with Interpolation
      6.4.2 利用插值法进行模仿
      • 6.4.2.1 Generator in the simulator
        6.4.2.1 模拟器中的发电机
      • 6.4.2.2 Downsampling of generated trajectories
        6.4.2.2 对生成的轨迹进行下采样
      • 6.4.2.3 Interpolation-Discriminator
        6.4.2.3 内插法-鉴别器
        • 6.4.2.3.1 Interpolator module
          6.4.2.3.1 内插模块
        • 6.4.2.3.2 Discriminator module
          6.4.2.3.2 鉴别器模块
        • 6.4.2.3.3 Loss function of Interpolation-Discriminator
          6.4.2.3.3 内插法-判别器的损失函数
    • 6.4.3 Training and Implementation
      6.4.3 培训和实施
  • 6.5 Experiment  6.5 实验
    • 6.5.1 Experimental Settings
      6.5.1 实验设置
      • 6.5.1.1 Dataset 6.5.1.1 数据集
      • 6.5.1.1 Synthetic Data  6.5.1.1 合成数据
        • 6.5.1.1.2 Real-world Data
          6.5.1.1.2 真实世界数据
      • 6.5.1.2 Data Preprocessing
        6.5.1.2 数据预处理
    • 6.5.2 Compared Methods  6.5.2 比较方法
      • 6.5.2.1 Calibration-based methods
        6.5.2.1 基于校准的方法
        6.5.2.2 Imitation learning-based methods
        6.5.2.2 基于模仿学习的方法

• 2.1 Definitions of traffic movement and traffic signal phases.
  2.1 交通流和交通信号阶段的定义。
• 2.2 RL framework for traffic signal control.
  2.2 交通信号控制的 RL 框架。
• 3.1 Reward is not a comprehensive measure to evaluate traffic light control performance. Both policies will lead to the same rewards. But policy #1 is more suitable than policy #2 in the real world.
  3.1 奖励不是评价交通灯控制性能的全面措施。两种政策会带来相同的奖励。但在现实世界中，1 号政策比 2 号政策更合适。
• 3.2 Case A and case B have the same environment except the traffic light phase.
  3.2 案例 A 和案例 B 的环境相同，但红绿灯阶段不同。
• 3.3 Model framework 3.3 模型框架
• 3.4 Q-network 3.4 Q 网络
• 3.5 Memory palace structure
  3.5 记忆宫殿结构
• 3.6 Traffic surveillance cameras in Jinan, China
  3.6 中国济南的交通监控摄像头
• 3.7 Percentage of the time duration of learned policy for phase Green-WE (green light on W-E and E-W direction, while red light on N-S and S-N direction) in every 2000 seconds for different methods under configuration 4.
  3.7 在配置 4 下，不同方法在绿-绿（W-E 和 E-W 方向亮绿灯，N-S 和 S-N 方向亮红灯）阶段每 2000 秒所学策略持续时间的百分比。
• 3.8 Average arrival rate on two directions (WE and SN) and time duration ratio of two phases (Green-WE and Red-WE) from learned policy for Jingliu Road (WE) and Erhuanxi Auxiliary Road (SN) in Jinan on August 1st and August 7th, 2016.
  3.8 2016 年 8 月 1 日和 8 月 7 日济南市经六路（WE）和二环西辅路（SN）的两个方向（WE 和 SN）的平均到达率和两个阶段（绿-WE 和红-WE）的时间长度比。

• 4.1 Performance of RL approaches is sensitive to reward and state. (a) A heuristic parameter tuning of reward function could result in different performances. (b) The method with a more complicated state (LIT[1] w/ neighbor) has a longer learning time but does not necessarily converge to a better result.
  4.1 RL 方法的性能对奖励和状态很敏感。(a) 奖赏函数的启发式参数调整可能导致不同的性能。(b) 状态更复杂的方法（LIT[1] w/ neighbor）学习时间更长，但不一定收敛到更好的结果。
• 4.2 Illustration of max pressure control in two cases. In Case A, green signal is set in the North$\to$$\to$rarr\rightarrow$\to$South direction; in Case B, green signal is set in the East$\to$$\to$rarr\rightarrow$\to$West direction.
  4.2 两种情况下的最大压力控制示意图。在情况 A 中，绿色信号设置在北 $\to$$\to$rarr\rightarrow$\to$ 向；在情况 B 中，绿色信号设置在东 $\to$$\to$rarr\rightarrow$\to$ 向。南方向；在情况 B 中，绿色信号设置在东 $\to$$\to$rarr\rightarrow$\to$ 西方向。西方向。
• 4.3 The transition of traffic movements.
  4.3 交通流的过渡。
• 4.4 Real-world arterial network for the experiment.
  4.4 用于实验的真实世界动脉网络。
• 4.5 Convergence curve of average duration and our reward design (pressure). Pressure shows the same convergence trend with travel time.
  4.5 平均持续时间与我们的奖励设计（压力）的收敛曲线。压力与旅行时间的收敛趋势相同。
• 4.6 Average travel time of our method on heterogeneous intersections. (a) Different number of legs. (b) Different length of lanes. (c) Experiment results.
  4.6 我们的方法在异构交叉口的平均行驶时间。(a) 不同的支腿数。 (b) 不同的车道长度。(c) 实验结果。
• 4.7 Performance comparison under uniform unidirectional traffic, where the optimal solution is known (GreenWave). Only PressLight can achieve the optimal.
  4.7 已知最优解（GreenWave）的均匀单向流量下的性能比较。只有 PressLight 可以达到最优。
• 4.8 Offsets between intersections learnt by RL agents under uni-directional uniform traffic (700 vehicles/hour/lane on arterial)
  4.8 在单向均匀交通（主干道上每小时每车道 700 辆车）条件下，RL 代理了解到的交叉口之间的偏移量
• 4.9 Space-time diagram with signal timing plan to illustrate the learned coordination strategy from real-world data on the arterial of Qingdao Road in the morning (around 8:30 a.m.) on August 6th.
  4.9 根据 8 月 6 日上午（8:30 左右）青岛路干道的实际数据绘制的时空图和信号配时方案，以说明所学的协调策略。

• 5.2 Left: Framework of the proposed CoLight model. Right: variation of cooperation scope (light blue shadow, from one-hop to two-hop) and attention distribution (colored points, the redder, the more important) of the target intersection.
  5.2 左图：建议的 CoLight 模型框架。右图：目标交叉点的合作范围（浅蓝色阴影，从一跳到两跳）和注意力分布（彩色点，越红越重要）的变化。
• 5.3 Road networks for real-world datasets. Red polygons are the areas we select to model, blue dots are the traffic signals we control. Left: 196 intersections with uni-directional traffic, middle: 16 intersections with uni-& bi-directional traffic, right: 12 intersections with bi-directional traffic..
  5.3 真实世界数据集的道路网络。红色多边形为我们选择的建模区域，蓝色圆点为我们控制的交通信号灯。左图：196 个单向交通路口，中图：16 个单向和双向交通路口，右图：12 个双向交通路口。
• 5.4 Convergence speed of CoLight (red continuous curves) and other 5 RL baselines (dashed curves) during training. CoLight starts with the best performance (Jumpstart), reaches to the pre-defined performance the fastest (Time to Threshold), and ends with the optimal policy (Aysmptotic). Curves are smoothed with a moving average of 5 points.
  5.4 在训练过程中，CoLight（红色连续曲线）和其他 5 条 RL 基线（虚线）的收敛速度。CoLight 以最佳性能（Jumpstart）开始，以最快速度达到预定性能（Time to Threshold），以最优策略（Aysmptotic）结束。曲线用 5 个点的移动平均值进行平滑处理。
• 5.5 The training time of different models for 100 episodes. CoLight is efficient across all the datasets. The bar for Individual RL on $D_{NewYork}$$D_{NewYork}$D_(NewYork)D_{NewYork}$D_{NewYork}$ is shadowed as its running time is far beyond the acceptable time.
  5.5 不同模型对 100 个事件的训练时间。CoLight 在所有数据集上都很高效。由于 $D_{NewYork}$$D_{NewYork}$D_(NewYork)D_{NewYork}$D_{NewYork}$ 上的 Individual RL 的运行时间远远超出了可接受的时间，因此其条形图上有阴影。
• 5.6 Performance of CoLight with respect to different numbers of neighbors ($|{\mathcal{N}}_i|$$|{\mathcal{N}}_i|$|N_(i)||\mathcal{N}_{i}|$|{\mathcal{N}}_i|$) on dataset $D_{Hangzhou}$$D_{Hangzhou}$D_(Hangzhou)D_{Hangzhou}$D_{Hangzhou}$ (left) and $D_{Jinan}$$D_{Jinan}$D_(Jinan)D_{Jinan}$D_{Jinan}$ (right). More neighbors ($|{\mathcal{N}}_i|\le 5$$|{\mathcal{N}}_i|\le 5$|N_(i)| <= 5|\mathcal{N}_{i}|\leq 5$|{\mathcal{N}}_i|\le 5$) for cooperation brings better performance, but too many neighbors ($|{\mathcal{N}}_i|>5$$|{\mathcal{N}}_i|>5$|N_(i)| > 5|\mathcal{N}_{i}|>5$|{\mathcal{N}}_i|>5$) requires more time (200 episodes or more) to learn..
  5.6 在数据集 $D_{Hangzhou}$$D_{Hangzhou}$D_(Hangzhou)D_{Hangzhou}$D_{Hangzhou}$ 和 $D_{Jinan}$$D_{Jinan}$D_(Jinan)D_{Jinan}$D_{Jinan}$ 中，CoLight 在不同邻接数（ $|{\mathcal{N}}_i|$$|{\mathcal{N}}_i|$|N_(i)||\mathcal{N}_{i}|$|{\mathcal{N}}_i|$ ）下的性能表现(左）和 $D_{Jinan}$$D_{Jinan}$D_(Jinan)D_{Jinan}$D_{Jinan}$ （右(右图）。更多的邻居（ $|{\mathcal{N}}_i|\le 5$$|{\mathcal{N}}_i|\le 5$|N_(i)| <= 5|\mathcal{N}_{i}|\leq 5$|{\mathcal{N}}_i|\le 5$ ）会带来更好的合作效果，但过多的邻居（ $|{\mathcal{N}}_i|>5$$|{\mathcal{N}}_i|>5$|N_(i)| > 5|\mathcal{N}_{i}|>5$|{\mathcal{N}}_i|>5$ ）则需要更多的时间（200 集或更多）来学习。
• 6.1 Illustration of a driving trajectory. In the real-world scenario, only part of the driving points can be observed and form a sparse driving trajectory (in red dots). Each driving point includes a driving state and an action of the vehicle at the observed time step. Best viewed in color.
  6.1 行驶轨迹示意图。在实际场景中，只有部分行驶点可以被观测到，并形成稀疏的行驶轨迹（红点）。每个行车点都包括观察到的时间步长内的行车状态和车辆动作。以彩色显示效果最佳。
• 6.2 Proposed ImIn-GAIL Approach. The overall framework of ImIn-GAIL includes three components: generator, downsampler, and interpolation-discriminator. Best viewed in color.
  6.2 拟议的 ImIn-GAIL 方法。ImIn-GAIL 的整体框架包括三个部分：生成器、下采样器和插值-鉴别器。彩色效果最佳。
• 6.3 Proposed interpolation-discriminator network.
  6.3 拟议的插值-鉴别器网络。
• 6.4 Illustration of road networks. (a) and (b) are synthetic road networks, while (c) and (d) are real-world road networks.
  6.4 道路网络示意图。(a) 和 (b) 是合成的道路网络，而 (c) 和 (d) 是真实世界的道路网络。
• 6.5RMSE on time and position of our proposed method ImIn-GAIL under different level of sparsity. As the expert trajectory become denser, a more similar policy to the expert policy is learned.
  6.5 我们提出的 ImIn-GAIL 方法在不同稀疏程度下的时间和位置均方根误差。随着专家轨迹越来越密集，学习到的策略与专家策略也越来越相似。
• 6.6 The generated trajectory of a vehicle in the $Ring$$Ring$RingRing$Ring$ scenario. Left: the initial position of the vehicles. Vehicles can only be observed when they pass four locations $A$$A$AA$A$, $B$$B$BB$B$, $C$$C$CC$C$ and $D$$D$DD$D$ where cameras are installed. Right: the visualization for the trajectory of $Vehicle$$Vehicle$VehicleVehicle$Vehicle$ 0. The x-axis is the timesteps in seconds. The y-axis is the relative road distance in meters. Although vehicle 0 is only observed three times (red triangles), ImIn-GAIL (blue points) can imitate the position of the expert trajectory (grey points) more accurately than all other baselines. Better viewed in color.
  6.6 在 $Ring$$Ring$RingRing$Ring$ 场景中生成的车辆轨迹。左图：车辆的初始位置。车辆只有在经过 $A$$A$AA$A$ 、 $B$$B$BB$B$ 和 $C$$C$CC$C$ 四个位置时才能被观测到。, $B$$B$BB$B$、$C$$C$CC$C$ 和 $D$$D$DD$D$ 安装了摄像头。右图： $Vehicle$$Vehicle$VehicleVehicle$Vehicle$ 0 的可视化轨迹。x 轴为时间单位，以秒为单位。y 轴为相对道路距离，单位为米。虽然只观察到车辆 0（红色三角形）三次，但 ImIn-GAIL（蓝色点）比其他所有基线都能更准确地模仿专家轨迹（灰色点）的位置。彩色效果更佳。

• 2.1 Representative deep RL-based traffic signal control methods. Due to page limits, we only put part of the investigated papers here.
  2.1 基于深度 RL 的代表性交通信号控制方法。由于篇幅限制，我们在此仅介绍部分研究论文。
• 3.1 Notations 3.1 符号
• 3.2 Settings for our method
  3.2 我们方法的设置
• 3.3 Reward coefficients 3.3 奖励系数
• 3.4 Configurations for synthetic traffic data
  3.4 合成流量数据的配置
• 3.5 Details of real-world traffic dataset
  3.5 真实世界交通数据集的细节
• 3.6 Performance on configuration 1. Reward: the higher the better. Other measures: the lower the better. Same with the following tables.
  3.6 配置 1 的性能。奖励：越高越好。其他措施：越低越好。与下表相同。
• 3.7 Performance on configuration 2
  3.7 配置 2 的性能
• 3.8 Performance on configuration 3
  3.8 配置 3 的性能
• 3.9 Performance on configuration 4
  3.9 配置 4 的性能
• 3.10 Performances of different methods on real-world data. The number after $\pm$$\pm$+-\pm$\pm$ means standard deviation. Reward: the higher the better. Other measures: the lower the better.
  3.10 不同方法在实际数据中的表现。 $\pm$$\pm$+-\pm$\pm$ 后面的数字表示标准偏差。奖励：越高越好。其他指标：越低越好。
• 4.1 Summary of notation.
  4.1 符号摘要
• 4.2 Configurations for synthetic traffic data
  4.2 合成流量数据的配置
• 4.3 Data statistics of real-world traffic dataset
  4.3 真实世界交通数据集的数据统计

• 4.5 Detailed comparison of our proposed state and reward design and their effects w.r.t. average travel time (lower the better) under synthetic traffic data.
  4.5 在合成交通数据下，详细比较我们建议的状态和奖励设计及其对平均旅行时间（越短越好）的影响。
• 4.6 Average travel time of different methods under arterials with a different number of intersections and network.
  4.6 在交叉口数量和网络不同的干道上，不同方法的平均旅行时间。
• 5.1 Data statistics of real-world traffic dataset
  5.1 真实世界交通数据集的数据统计
• 5.2 Performance on synthetic data and real-world data w.r.t average travel time. CoLight is the best.
  5.2 平均旅行时间在合成数据和真实世界数据上的表现。CoLight 是最好的。
• 5.3 Performance of CoLight with respect to different numbers of attention heads ($H$$H$HH$H$) on dataset $Gri{d}_{6\times 6}$$Gri{d}_{6\times 6}$Grid_(6xx6)Grid_{6\times 6}$Gri{d}_{6\times 6}$. More types of attention ($H\le 5$$H\le 5$H <= 5H\leq 5$H\le 5$) enhance model efficiency, while too many ($H>5$$H>5$H > 5H>5$H>5$) could distract the learning and deteriorate the overall performance.
  5.3 在数据集 $Gri{d}_{6\times 6}$$Gri{d}_{6\times 6}$Grid_(6xx6)Grid_{6\times 6}$Gri{d}_{6\times 6}$ 中，CoLight 在不同注意力头数（ $H$$H$HH$H$ ）下的性能表现.更多的注意力类型（ $H\le 5$$H\le 5$H <= 5H\leq 5$H\le 5$ ）会提高模型效率，而过多的注意力类型（ $H>5$$H>5$H > 5H>5$H>5$ ）则会分散学习注意力，降低整体性能。
• 6.1 Features for a driving state
  6.1 驾驶状态的特征
• 6.2 Hyper-parameter settings for ImIn-GAIL
  6.2 ImIn-GAIL 的超参数设置
• 6.3 Statistics of dense and sparse expert trajectory in different datasets
  6.3 不同数据集中密集和稀疏专家轨迹的统计数据
• 6.4 Performance w.r.t Relative Mean Squared Error (RMSE) of time (in seconds) and position (in kilometers). All the measurements are conducted on dense trajectories. Lower the better. Our proposed method ImIn-GAIL achieves the best performance.
  6.4 时间（秒）和位置（千米）的相对均方差（RMSE）性能。所有测量均在密集轨迹上进行。越低越好。我们提出的 ImIn-GAIL 方法性能最佳。
• 6.5 RMSE on time and position of our proposed method ImIn-GAIL against baseline methods and their corresponding two-step variants. Baseline methods and ImIn-GAIL learn from sparse trajectories, while the two-step variants interpolate sparse trajectories first and trained on the interpolated data. ImIn-GAIL achieves the best performance in most cases.
  6.5 与基准方法及其相应的两步变体相比，我们提出的 ImIn-GAIL 方法在时间和位置上的均方根误差。基准方法和 ImIn-GAIL 从稀疏轨迹中学习，而两步变体首先对稀疏轨迹进行插值，然后在插值数据上进行训练。在大多数情况下，ImIn-GAIL 的性能最佳。

Acknowledgements 致谢

Chapter 1 Introduction 第 1 章 导言

1.1 Why do we need a more intelligent traffic signal control
1.1 为什么我们需要更智能的交通信号控制？

1.2 Why do we use reinforcement learning for traffic signal control
1.2 为什么要将强化学习用于交通信号控制

1.3 Why RL for traffic signal control is challenging?
1.3 交通信号控制 RL 为何具有挑战性？

Formulation of RL agent
RL 剂的配方

Learning cost 学习成本

Simulation 模拟

1.5 Proposed Tasks 1.5 拟议任务

• Formulation of RL with theoretical justification. How to formulate the RL agent, i.e., the reward and state definition for traffic signal control problem? Does it always beneficial if we use complex reward function and state representations? In [7, 1, 17], I proposed to deduce the intricate design of current literature and use concise reward and state design, with theoretical proof on the concise design towards the global optimal, for both signal intersection control and multi-intersection control scenario.
  制定 RL 的理论依据。如何制定 RL 代理，即交通信号控制问题的奖励和状态定义？使用复杂的奖励函数和状态表示是否一定有益？在文献[7, 1, 17]中，笔者提出对现有文献的复杂设计进行演绎，使用简明的奖励和状态设计，并对信号交叉口控制和多交叉口控制场景下的简明设计进行了全局最优的理论证明。
• Optimization of the learning process in RL. Is vanilla deep neural network the solution to traffic signal control problem with RL? In [11], I tackle the sample imbalance problem with a phase-gated network to emphasize certain features. In [18], we design a network to model the priority between action and learn the change of traffic flow like flip and rotation. Coordination could benefit signal control for multi-intersection scenarios. In [19], I contributed a framework to enable agents to communicate between them about their observations and behave as a group. Though each agent has limited capabilities and visibility of the world, in this work, agents were able to cooperate multi-hop neighboring intersections and learn the dynamic interactions between the hidden state of neighboring agents. Later in [20], we investigate the possibility of using RL to control traffic signals under the scale of a city, test our RL methods in the simulator under the road network of Manhattan, New York, with 2510 traffic signals. This is the first time an RL-based method can operate on such a scale in traffic signal control.
  优化 RL 的学习过程。香草深度神经网络是利用 RL 解决交通信号控制问题的方法吗？在 [11] 中，我用相位门控网络来解决样本不平衡问题，以强调某些特征。在文献[18]中，我们设计了一个网络来模拟行动之间的优先级，并学习交通流的变化，如翻转和旋转。在多交叉口情况下，协调可有利于信号控制。在文献[19]中，我提出了一个框架，使代理之间能够就其观察结果进行交流，并作为一个群体行事。虽然每个代理的能力和对世界的可见度都有限，但在这项工作中，代理能够在相邻交叉口进行多跳合作，并学习相邻代理隐藏状态之间的动态交互。随后在 [20] 中，我们研究了在城市规模下使用 RL 控制交通信号的可能性，在纽约曼哈顿拥有 2510 个交通信号的道路网络下的模拟器中测试了我们的 RL 方法。这是基于 RL 的方法首次在如此大规模的交通信号控制中发挥作用。
• Bridging Gap from Simulation to Reality. Despite its massive success in artificial domains, RL has not yet shown the same degree of success for real-world applications. These applications could involve control systems such as drones, software systems such as data centers, systems that interact with people such as transportation systems. Currently, most RL methods mainly conduct experiments in the simulator since the simulator can generate data in a cheaper and faster way than real experimentation. Discrepancies between simulation and reality confine the application of learned policies in the real world.
  缩小模拟与现实之间的差距。尽管 RL 在人工智能领域取得了巨大成功，但在现实世界的应用中还没有取得同样的成功。这些应用可能涉及无人机等控制系统、数据中心等软件系统、交通系统等与人交互的系统。目前，大多数 RL 方法主要在模拟器中进行实验，因为模拟器生成数据的方式比真实实验更便宜、更快捷。模拟与现实之间的差异限制了所学政策在现实世界中的应用。

1.6 Overview of this Disseration
1.6 本论文概述

Chapter 2 Notation, Background and Literature
第 2 章 符号、背景和文献

2.1 Preliminaries of Traffic Signal Control Problem
2.1 交通信号控制问题初探

Term Definition 术语定义

• Approach: A roadway meeting at an intersection is referred to as an approach. At any general intersection, there are two kinds of approaches: incoming approaches and outgoing approaches. An incoming approach is one on which cars can enter the intersection; an outgoing approach is one on which cars can leave the intersection. Figure 2.1(a) shows a typical intersection with four incoming approaches and outgoing approaches. The southbound incoming approach is denoted in this figure as the approach on the north side in which vehicles are traveling in the southbound direction.
  引道：在交叉路口交汇的道路称为引道。在任何一个普通交叉路口，都有两种引道：进入引道和驶出引道。进路是指车辆可以进入交叉路口的道路；出路是指车辆可以离开交叉路口的道路。图 2.1(a) 显示了一个有四条进路和出路的典型交叉路口。南行进路在本图中表示为车辆南行的北侧进路。
• Lane: An approach consists of a set of lanes. Similar to approach definition, there are two kinds of lanes: incoming lanes and outgoing lanes (also known as approaching/entering lane and receiving/exiting lane in some references [23, 24]).
  车道：航道由一组车道组成。与路径定义类似，车道也分为两种：进入车道和离开车道（在一些参考文献中也称为接近/进入车道和接收/离开车道[23, 24]）。
• Traffic movement: A traffic movement refers to vehicles moving from an incoming approach to an outgoing approach, denoted as $(r_i\to r_o)$$(r_i\to r_o)$(r_(i)rarrr_(o))(r_{i}\to r_{o})$(r_i\to r_o)$, where $r_a$$r_a$r_(a)r_{a}$r_a$ and $r_r$$r_r$r_(r)r_{r}$r_r$ is theincoming lane and the outgoing lane respectively. A traffic movement is generally categorized as left turn, through, and right turn.
  交通流：交通移动是指车辆从进站通道驶向出站通道，表示为 $(r_i\to r_o)$$(r_i\to r_o)$(r_(i)rarrr_(o))(r_{i}\to r_{o})$(r_i\to r_o)$ ，其中 $r_a$$r_a$r_(a)r_{a}$r_a$ 和 $r_r$$r_r$r_(r)r_{r}$r_r$ 分别为进站车道和出站车道。其中，$r_a$$r_a$r_(a)r_{a}$r_a$ 和 $r_r$$r_r$r_(r)r_{r}$r_r$ 分别为进入车道和驶出车道。交通流一般分为左转、通过和右转。

• Movement signal: A movement signal is defined on the traffic movement, with green signal indicating the corresponding movement is allowed and red signal indicating the movement is prohibited. For the four-leg intersection shown in Figure 2.1(a), the right-turn traffic can pass regardless of the signal, and there are eight movement signals in use, as shown in Figure 2.1(b).
  移动信号：移动信号：移动信号定义在交通移动上，绿灯表示允许相应移动，红灯表示禁止移动。对于图 2.1(a)所示的四脚交叉路口，无论信号灯如何，右转车辆均可通过，如图 2.1(b)所示，共有八个移动信号灯在使用。
• Phase: A phase is a combination of movement signals. Figure 2.1(c) shows the conflict matrix of the combination of two movement signals in the example in Figure 2.1(a) and Figure 2.1(b). The grey cell indicates the corresponding two movements conflict with each other, i.e. they cannot be set to 'green' at the same time (e.g., signals #1 and #2). The white cell indicates the non-conflicting movement signals. All the non-conflicting signals will generate eight valid paired-signal phases (letters 'A' to 'H' in Figure 2.1(c)) and eight single-signal phases (the diagonal cells in conflict matrix). Here we letter the paired-signal phases only because in an isolated intersection, it is always more efficient to use paired-signal
  相位：相位：相位是运动信号的组合。图 2.1(c)显示了图 2.1(a)和图 2.1(b)中两个运动信号组合的冲突矩阵。灰色单元格表示相应的两个运动信号相互冲突，即不能同时设置为 "绿色"（如 1 号和 2 号信号）。白色单元格表示不冲突的运动信号。所有非冲突信号将产生八个有效的成对信号相位（图 2.1(c)中的字母 "A "至 "H"）和八个单信号相位（冲突矩阵中的对角线单元格）。在这里，我们只提及成对信号相位，因为在一个孤立的交叉路口，使用成对信号相位总是更有效率。

• Phase sequence: A phase sequence is a sequence of phases which defines a set of phases and their order of changes.
  相序：阶段序列是一个阶段序列，它定义了一组阶段及其变化顺序。
• Signal plan: A signal plan for a single intersection is a sequence of phases and their corresponding starting time. Here we denote a signal plan as $(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$(p_(1),t_(1))(p_(2),t_(2))dots(p_(i),t_(i))dots(p_{1},t_{1})(p_{2},t_{2})\dots(p_{i},t_{i})\dots$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$, where $p_i$$p_i$p_(i)p_{i}$p_i$ and $t_i$$t_i$t_(i)t_{i}$t_i$ stand for a phase and its starting time.
  信号计划：单个交叉口的信号计划是一系列相位及其相应的起始时间。在此，我们用 $(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$(p_(1),t_(1))(p_(2),t_(2))dots(p_(i),t_(i))dots(p_{1},t_{1})(p_{2},t_{2})\dots(p_{i},t_{i})\dots$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$ 表示一个信号计划。其中 $p_i$$p_i$p_(i)p_{i}$p_i$ 和 $t_i$$t_i$t_(i)t_{i}$t_i$ 代表一个相位及其起始时间。
• Cycle-based signal plan: A cycle-based signal plan is a kind of signal plan where the sequence of phases operates in a cyclic order, which can be denoted as $(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$(p_(1),t_(1)^(1))(p_(2),t_(2)^(1))dots(p_(N),t_(N)^(1))(p_(1),t_(1)^(2))(p_(2),t_(2)^(2))dots(p_(N),t_(N)^(2))dots(p_{1},t_{1}^{1})(p_{2},t_{2}^{1})\dots(p_{N},t_{N}^{1})(p_{1},t_{1}^{2})(p_{ 2},t_{2}^{2})\dots(p_{N},t_{N}^{2})\dots$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$, where $p_1,p_2,\dots ,p_N$$p_1,p_2,\dots ,p_N$p_(1),p_(2),dots,p_(N)p_{1},p_{2},\dots,p_{N}$p_1,p_2,\dots ,p_N$ is the repeated phase sequence and $t_i^j$$t_i^j$t_(i)^(j)t_{i}^{j}$t_i^j$ is the starting time of phase $p_i$$p_i$p_(i)p_{i}$p_i$ in the $j$$j$jj$j$-th cycle. Specifically, $C^j=t_1^{j+1}-t_1^j$$C^j=t_1^{j+1}-t_1^j$C^(j)=t_(1)^(j+1)-t_(1)^(j)C^{j}=t_{1}^{j+1}-t_{1}^{j}$C^j=t_1^{j+1}-t_1^j$ is the cycle length of the $j$$j$jj$j$-th phase cycle, and $\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}${(t_(2)^(j)-t_(1)^(j))/(C^(j)),dots,(t_(N)^(j)-t_(N-1)^(j))/(C^(j))}\{\frac{t_{2}^{j}-t_{1}^{j}}{C^{j}},\dots,\frac{t_{N}^{j}-t_{N-1}^{j}}{C^{j}}\}$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$ is the phase split of the $j$$j$jj$j$-th phase cycle. Existing traffic signal control methods usually repeats similar phase sequence throughout the day.
  循环信号计划：循环信号计划是一种相位顺序以循环顺序运行的信号计划，可表示为 $(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$(p_(1),t_(1)^(1))(p_(2),t_(2)^(1))dots(p_(N),t_(N)^(1))(p_(1),t_(1)^(2))(p_(2),t_(2)^(2))dots(p_(N),t_(N)^(2))dots(p_{1},t_{1}^{1})(p_{2},t_{2}^{1})\dots(p_{N},t_{N}^{1})(p_{1},t_{1}^{2})(p_{ 2},t_{2}^{2})\dots(p_{N},t_{N}^{2})\dots$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$ ，其中 $p_1,p_2,\dots ,p_N$$p_1,p_2,\dots ,p_N$p_(1),p_(2),dots,p_(N)p_{1},p_{2},\dots,p_{N}$p_1,p_2,\dots ,p_N$ 为重复相位顺序， $t_i^j$$t_i^j$t_(i)^(j)t_{i}^{j}$t_i^j$ 为 $j$$j$jj$j$ 中相位 $p_i$$p_i$p_(i)p_{i}$p_i$ 的起始时间。其中，$p_1,p_2,\dots ,p_N$$p_1,p_2,\dots ,p_N$p_(1),p_(2),dots,p_(N)p_{1},p_{2},\dots,p_{N}$p_1,p_2,\dots ,p_N$ 为重复的相位序列， $t_i^j$$t_i^j$t_(i)^(j)t_{i}^{j}$t_i^j$ 为相位$p_i$$p_i$p_(i)p_{i}$p_i$ 在$j$$j$jj$j$ -次循环中的起始时间。-的开始时间。具体来说， $C^j=t_1^{j+1}-t_1^j$$C^j=t_1^{j+1}-t_1^j$C^(j)=t_(1)^(j+1)-t_(1)^(j)C^{j}=t_{1}^{j+1}-t_{1}^{j}$C^j=t_1^{j+1}-t_1^j$ 是 $j$$j$jj$j$ -次相位周期的周期长度， $\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}${(t_(2)^(j)-t_(1)^(j))/(C^(j)),dots,(t_(N)^(j)-t_(N-1)^(j))/(C^(j))}\{\frac{t_{2}^{j}-t_{1}^{j}}{C^{j}},\dots,\frac{t_{N}^{j}-t_{N-1}^{j}}{C^{j}}\}$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$ 是{{8} -次相位周期的周期长度。的周期长度， $\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}${(t_(2)^(j)-t_(1)^(j))/(C^(j)),dots,(t_(N)^(j)-t_(N-1)^(j))/(C^(j))}\{\frac{t_{2}^{j}-t_{1}^{j}}{C^{j}},\dots,\frac{t_{N}^{j}-t_{N-1}^{j}}{C^{j}}\}$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$ 是 $j$$j$jj$j$ -个周期的相位分割。-第 4 个相位周期的相位分割。现有的交通信号控制方法通常全天重复类似的相位顺序。

Objective 目标

• Travel time. In traffic signal control, travel time of a vehicle is defined as the time different between the time one car enters the system and the time it leaves the system. One of the most common goals is to minimize the average travel time of vehicles in the network.
  行驶时间。在交通信号控制中，车辆的行驶时间被定义为一辆车从进入系统到离开系统之间的时间差。最常见的目标之一就是尽量缩短网络中车辆的平均行驶时间。
• Queue length. The queue length of the road network is the number of queuing vehicles in the road network.
  队列长度。路网的队列长度是指路网中排队车辆的数量。
• Number of stops. The number of stops of a vehicle is the total times that a vehicle experienced.
  停车次数。车辆的停车次数是指车辆经历的总次数。
• Throughput. The throughput is the number of vehicles that have completed their trip in the road network during throughout a period.
  吞吐量。吞吐量是指在整个时段内完成路网行程的车辆数量。

Special Considerations 特殊考虑因素

1. Yellow and all-red time. A yellow signal is usually set as a transition from a green signal to a red one. Following the yellow, there is an all-red period during which all the signals in an intersection are set to red. The yellow and all-red time, which can last from 3 to 6 seconds, allow vehicles to stop safely or pass the intersection before vehicles in conflicting traffic movements are given a green signal.
   黄色和全红时间。黄色信号灯通常是从绿色信号灯向红色信号灯的过渡。黄灯之后是全红灯时间，在此期间，交叉路口的所有信号灯都设置为红灯。黄色信号灯和全红信号灯时间可持续 3 到 6 秒钟，这段时间允许车辆安全停靠或通过交叉路口，然后再向与之交通流冲突的车辆发出绿色信号灯。
2. Minimum green time. Usually, a minimum green signal time is required to ensure pedestrians moving during a particular phase can safely pass through the intersection.
   最短绿灯时间。通常情况下，需要一个最短的绿灯时间，以确保在某一特定阶段通行的行人能安全通过交叉路口。
3. Left turn phase. Usually, a left-turn phase is added when the left-turn volume is above certain threshold.
   左转阶段。通常，当左转量超过一定临界值时，就会增加一个左转阶段。

2.2 Background of Reinforcement Learning
2.2 强化学习的背景

Single Agent Reinforcement learning
单个代理强化学习

Problem setting 问题设置

2.3 Literature 2.3 文献资料

Agent formulation 制剂配方

2.3.1.1 Reward 2.3.1.1 奖励

2.3.1.2 State 2.3.1.2 国家

2.3.1.3 Action scheme 2.3.1.3 行动方案

Policy Learning 政策学习

2.3.2.1 Value-based methods
2.3.2.1 基于价值的方法

Joint action learners 联合行动学习者