deep_reinforcement_learning_fo-183ae653-e302-46bb-993b-53ac63dce2ed

Abstract

Table of Contents

Acknowledgements

Chapter 2 Notation, Background and Literature

• 2.4 Conclusion

Chapter 4 Formulating the Learning Objevtive

* 4.1 Introduction
* 4.2 Related Work
* 4.3 Preliminaries and Notations
* 4.4 Method
	* 4.4.1 Agent Design
	* 4.4.2 Learning Process
* 4.5 Justification of RL agent
	* 4.5.1 Justification for State Design
		* 4.5.1.1 General description of traffic movement process as a Markov chain
		* 4.5.1.2 Specification with proposed state definition
	* 4.5.2 Justification for Reward Design
		* 4.5.2.1 Stabilization on traffic movements with proposed reward.
		* 4.5.2.2 Connection to throughput maximization and travel time minimization.
* 4.6 Experiment
	* 4.6.1 Dataset Description
	* 4.6.2 Experimental Settings
		* 4.6.2.1 Environmental settings
		* 4.6.2.2 Evaluation metric
		* 4.6.2.3 Compared methods
	* 4.6.3 Performance Comparison
	* 4.6.4 Study of PressLight
		* 4.6.4.1 Effects of variants of our proposed method
		* 4.6.4.2 Average travel time related to pressure.
	* 4.6.5 Performance on Mixed Scenarios
		* 4.6.5.1 Heterogeneous intersections
		* 4.6.5.2 Arterials with a different number of intersections and network
	* 4.6.6 Case Study
		* 4.6.6.1 Synthetic traffic on the uniform, uni-directional flow
			* 4.6.6.1.1 Performance comparison
			* 4.6.6.1.2 Policy learned by RL agents
		* 4.6.6.2 Real-world traffic in Jinan
* 4.7 Conclusion

* 5.1 Introduction
* 5.2 Related Work
* 5.3 Problem Definition
* 5.4 Method
	* 5.4.1 Observation Embedding
	* 5.4.2 Graph Attention Networks for Cooperation
		* 5.4.2.1 Observation Interaction
		* 5.4.2.2 Attention Distribution within Neighborhood Scope
		* 5.4.2.3 Index-free Neighborhood Cooperation
		* 5.4.2.4 Multi-head Attention
	* 5.4.3 Q-value Prediction
	* 5.4.4 Complexity Analysis

Chapter 6 Learning to Simulate

• 6.1 Introduction
• 6.2 Related Work
• 6.3 Preliminaries
  • 6.4 Method
    • 6.4.1 Basic GAIL Framework
    • 6.4.2 Imitation with Interpolation
      • 6.4.2.1 Generator in the simulator
      • 6.4.2.2 Downsampling of generated trajectories
      • 6.4.2.3 Interpolation-Discriminator
        • 6.4.2.3.1 Interpolator module
        • 6.4.2.3.2 Discriminator module
        • 6.4.2.3.3 Loss function of Interpolation-Discriminator
    • 6.4.3 Training and Implementation
  • 6.5 Experiment
    • 6.5.1 Experimental Settings
      • 6.5.1.1 Dataset
      • 6.5.1.1 Synthetic Data
        • 6.5.1.1.2 Real-world Data
      • 6.5.1.2 Data Preprocessing
    • 6.5.2 Compared Methods
      • 6.5.2.1 Calibration-based methods
        6.5.2.2 Imitation learning-based methods

• 2.1 Definitions of traffic movement and traffic signal phases.
• 2.2 RL framework for traffic signal control.
• 3.1 Reward is not a comprehensive measure to evaluate traffic light control performance. Both policies will lead to the same rewards. But policy #1 is more suitable than policy #2 in the real world.
• 3.2 Case A and case B have the same environment except the traffic light phase.
• 3.3 Model framework
• 3.4 Q-network
• 3.5 Memory palace structure
• 3.6 Traffic surveillance cameras in Jinan, China
• 3.7 Percentage of the time duration of learned policy for phase Green-WE (green light on W-E and E-W direction, while red light on N-S and S-N direction) in every 2000 seconds for different methods under configuration 4.
• 3.8 Average arrival rate on two directions (WE and SN) and time duration ratio of two phases (Green-WE and Red-WE) from learned policy for Jingliu Road (WE) and Erhuanxi Auxiliary Road (SN) in Jinan on August 1st and August 7th, 2016.

• 4.1 Performance of RL approaches is sensitive to reward and state. (a) A heuristic parameter tuning of reward function could result in different performances. (b) The method with a more complicated state (LIT[1] w/ neighbor) has a longer learning time but does not necessarily converge to a better result.
• 4.2 Illustration of max pressure control in two cases. In Case A, green signal is set in the North$\to$$\to$rarr\rightarrow$\to$South direction; in Case B, green signal is set in the East$\to$$\to$rarr\rightarrow$\to$West direction.
• 4.3 The transition of traffic movements.
• 4.4 Real-world arterial network for the experiment.
• 4.5 Convergence curve of average duration and our reward design (pressure). Pressure shows the same convergence trend with travel time.
• 4.6 Average travel time of our method on heterogeneous intersections. (a) Different number of legs. (b) Different length of lanes. (c) Experiment results.
• 4.7 Performance comparison under uniform unidirectional traffic, where the optimal solution is known (GreenWave). Only PressLight can achieve the optimal.
• 4.8 Offsets between intersections learnt by RL agents under uni-directional uniform traffic (700 vehicles/hour/lane on arterial)
• 4.9 Space-time diagram with signal timing plan to illustrate the learned coordination strategy from real-world data on the arterial of Qingdao Road in the morning (around 8:30 a.m.) on August 6th.

• 5.2 Left: Framework of the proposed CoLight model. Right: variation of cooperation scope (light blue shadow, from one-hop to two-hop) and attention distribution (colored points, the redder, the more important) of the target intersection.
• 5.3 Road networks for real-world datasets. Red polygons are the areas we select to model, blue dots are the traffic signals we control. Left: 196 intersections with uni-directional traffic, middle: 16 intersections with uni-& bi-directional traffic, right: 12 intersections with bi-directional traffic..
• 5.4 Convergence speed of CoLight (red continuous curves) and other 5 RL baselines (dashed curves) during training. CoLight starts with the best performance (Jumpstart), reaches to the pre-defined performance the fastest (Time to Threshold), and ends with the optimal policy (Aysmptotic). Curves are smoothed with a moving average of 5 points.
• 5.5 The training time of different models for 100 episodes. CoLight is efficient across all the datasets. The bar for Individual RL on $D_{NewYork}$$D_{NewYork}$D_(NewYork)D_{NewYork}$D_{NewYork}$ is shadowed as its running time is far beyond the acceptable time.
• 5.6 Performance of CoLight with respect to different numbers of neighbors ($|{\mathcal{N}}_i|$$|{\mathcal{N}}_i|$|N_(i)||\mathcal{N}_{i}|$|{\mathcal{N}}_i|$) on dataset $D_{Hangzhou}$$D_{Hangzhou}$D_(Hangzhou)D_{Hangzhou}$D_{Hangzhou}$ (left) and $D_{Jinan}$$D_{Jinan}$D_(Jinan)D_{Jinan}$D_{Jinan}$ (right). More neighbors ($|{\mathcal{N}}_i|\le 5$$|{\mathcal{N}}_i|\le 5$|N_(i)| <= 5|\mathcal{N}_{i}|\leq 5$|{\mathcal{N}}_i|\le 5$) for cooperation brings better performance, but too many neighbors ($|{\mathcal{N}}_i|>5$$|{\mathcal{N}}_i|>5$|N_(i)| > 5|\mathcal{N}_{i}|>5$|{\mathcal{N}}_i|>5$) requires more time (200 episodes or more) to learn..
• 6.1 Illustration of a driving trajectory. In the real-world scenario, only part of the driving points can be observed and form a sparse driving trajectory (in red dots). Each driving point includes a driving state and an action of the vehicle at the observed time step. Best viewed in color.
• 6.2 Proposed ImIn-GAIL Approach. The overall framework of ImIn-GAIL includes three components: generator, downsampler, and interpolation-discriminator. Best viewed in color.
• 6.3 Proposed interpolation-discriminator network.
• 6.4 Illustration of road networks. (a) and (b) are synthetic road networks, while (c) and (d) are real-world road networks.
• 6.5RMSE on time and position of our proposed method ImIn-GAIL under different level of sparsity. As the expert trajectory become denser, a more similar policy to the expert policy is learned.
• 6.6 The generated trajectory of a vehicle in the $Ring$$Ring$RingRing$Ring$ scenario. Left: the initial position of the vehicles. Vehicles can only be observed when they pass four locations $A$$A$AA$A$, $B$$B$BB$B$, $C$$C$CC$C$ and $D$$D$DD$D$ where cameras are installed. Right: the visualization for the trajectory of $Vehicle$$Vehicle$VehicleVehicle$Vehicle$ 0. The x-axis is the timesteps in seconds. The y-axis is the relative road distance in meters. Although vehicle 0 is only observed three times (red triangles), ImIn-GAIL (blue points) can imitate the position of the expert trajectory (grey points) more accurately than all other baselines. Better viewed in color.

• 2.1 Representative deep RL-based traffic signal control methods. Due to page limits, we only put part of the investigated papers here.
• 3.1 Notations
• 3.2 Settings for our method
• 3.3 Reward coefficients
• 3.4 Configurations for synthetic traffic data
• 3.5 Details of real-world traffic dataset
• 3.6 Performance on configuration 1. Reward: the higher the better. Other measures: the lower the better. Same with the following tables.
• 3.7 Performance on configuration 2
• 3.8 Performance on configuration 3
• 3.9 Performance on configuration 4
• 3.10 Performances of different methods on real-world data. The number after $\pm$$\pm$+-\pm$\pm$ means standard deviation. Reward: the higher the better. Other measures: the lower the better.
• 4.1 Summary of notation.
• 4.2 Configurations for synthetic traffic data
• 4.3 Data statistics of real-world traffic dataset

• 4.5 Detailed comparison of our proposed state and reward design and their effects w.r.t. average travel time (lower the better) under synthetic traffic data.
• 4.6 Average travel time of different methods under arterials with a different number of intersections and network.
• 5.1 Data statistics of real-world traffic dataset
• 5.2 Performance on synthetic data and real-world data w.r.t average travel time. CoLight is the best.
• 5.3 Performance of CoLight with respect to different numbers of attention heads ($H$$H$HH$H$) on dataset $Gri{d}_{6\times 6}$$Gri{d}_{6\times 6}$Grid_(6xx6)Grid_{6\times 6}$Gri{d}_{6\times 6}$. More types of attention ($H\le 5$$H\le 5$H <= 5H\leq 5$H\le 5$) enhance model efficiency, while too many ($H>5$$H>5$H > 5H>5$H>5$) could distract the learning and deteriorate the overall performance.
• 6.1 Features for a driving state
• 6.2 Hyper-parameter settings for ImIn-GAIL
• 6.3 Statistics of dense and sparse expert trajectory in different datasets
• 6.4 Performance w.r.t Relative Mean Squared Error (RMSE) of time (in seconds) and position (in kilometers). All the measurements are conducted on dense trajectories. Lower the better. Our proposed method ImIn-GAIL achieves the best performance.
• 6.5 RMSE on time and position of our proposed method ImIn-GAIL against baseline methods and their corresponding two-step variants. Baseline methods and ImIn-GAIL learn from sparse trajectories, while the two-step variants interpolate sparse trajectories first and trained on the interpolated data. ImIn-GAIL achieves the best performance in most cases.

Acknowledgements

Chapter 1 Introduction

1.1 Why do we need a more intelligent traffic signal control

1.2 Why do we use reinforcement learning for traffic signal control

1.3 Why RL for traffic signal control is challenging?

Formulation of RL agent

Learning cost

Simulation

1.5 Proposed Tasks

• Formulation of RL with theoretical justification. How to formulate the RL agent, i.e., the reward and state definition for traffic signal control problem? Does it always beneficial if we use complex reward function and state representations? In [7, 1, 17], I proposed to deduce the intricate design of current literature and use concise reward and state design, with theoretical proof on the concise design towards the global optimal, for both signal intersection control and multi-intersection control scenario.
• Optimization of the learning process in RL. Is vanilla deep neural network the solution to traffic signal control problem with RL? In [11], I tackle the sample imbalance problem with a phase-gated network to emphasize certain features. In [18], we design a network to model the priority between action and learn the change of traffic flow like flip and rotation. Coordination could benefit signal control for multi-intersection scenarios. In [19], I contributed a framework to enable agents to communicate between them about their observations and behave as a group. Though each agent has limited capabilities and visibility of the world, in this work, agents were able to cooperate multi-hop neighboring intersections and learn the dynamic interactions between the hidden state of neighboring agents. Later in [20], we investigate the possibility of using RL to control traffic signals under the scale of a city, test our RL methods in the simulator under the road network of Manhattan, New York, with 2510 traffic signals. This is the first time an RL-based method can operate on such a scale in traffic signal control.
• Bridging Gap from Simulation to Reality. Despite its massive success in artificial domains, RL has not yet shown the same degree of success for real-world applications. These applications could involve control systems such as drones, software systems such as data centers, systems that interact with people such as transportation systems. Currently, most RL methods mainly conduct experiments in the simulator since the simulator can generate data in a cheaper and faster way than real experimentation. Discrepancies between simulation and reality confine the application of learned policies in the real world.

1.6 Overview of this Disseration

Chapter 2 Notation, Background and Literature

2.1 Preliminaries of Traffic Signal Control Problem

Term Definition

• Approach: A roadway meeting at an intersection is referred to as an approach. At any general intersection, there are two kinds of approaches: incoming approaches and outgoing approaches. An incoming approach is one on which cars can enter the intersection; an outgoing approach is one on which cars can leave the intersection. Figure 2.1(a) shows a typical intersection with four incoming approaches and outgoing approaches. The southbound incoming approach is denoted in this figure as the approach on the north side in which vehicles are traveling in the southbound direction.
• Lane: An approach consists of a set of lanes. Similar to approach definition, there are two kinds of lanes: incoming lanes and outgoing lanes (also known as approaching/entering lane and receiving/exiting lane in some references [23, 24]).
• Traffic movement: A traffic movement refers to vehicles moving from an incoming approach to an outgoing approach, denoted as $(r_i\to r_o)$$(r_i\to r_o)$(r_(i)rarrr_(o))(r_{i}\to r_{o})$(r_i\to r_o)$, where $r_a$$r_a$r_(a)r_{a}$r_a$ and $r_r$$r_r$r_(r)r_{r}$r_r$ is theincoming lane and the outgoing lane respectively. A traffic movement is generally categorized as left turn, through, and right turn.

• Movement signal: A movement signal is defined on the traffic movement, with green signal indicating the corresponding movement is allowed and red signal indicating the movement is prohibited. For the four-leg intersection shown in Figure 2.1(a), the right-turn traffic can pass regardless of the signal, and there are eight movement signals in use, as shown in Figure 2.1(b).
• Phase: A phase is a combination of movement signals. Figure 2.1(c) shows the conflict matrix of the combination of two movement signals in the example in Figure 2.1(a) and Figure 2.1(b). The grey cell indicates the corresponding two movements conflict with each other, i.e. they cannot be set to 'green' at the same time (e.g., signals #1 and #2). The white cell indicates the non-conflicting movement signals. All the non-conflicting signals will generate eight valid paired-signal phases (letters 'A' to 'H' in Figure 2.1(c)) and eight single-signal phases (the diagonal cells in conflict matrix). Here we letter the paired-signal phases only because in an isolated intersection, it is always more efficient to use paired-signal

• Phase sequence: A phase sequence is a sequence of phases which defines a set of phases and their order of changes.
• Signal plan: A signal plan for a single intersection is a sequence of phases and their corresponding starting time. Here we denote a signal plan as $(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$(p_(1),t_(1))(p_(2),t_(2))dots(p_(i),t_(i))dots(p_{1},t_{1})(p_{2},t_{2})\dots(p_{i},t_{i})\dots$(p_1,t_1)(p_2,t_2)\dots (p_i,t_i)\dots$, where $p_i$$p_i$p_(i)p_{i}$p_i$ and $t_i$$t_i$t_(i)t_{i}$t_i$ stand for a phase and its starting time.
• Cycle-based signal plan: A cycle-based signal plan is a kind of signal plan where the sequence of phases operates in a cyclic order, which can be denoted as $(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$(p_(1),t_(1)^(1))(p_(2),t_(2)^(1))dots(p_(N),t_(N)^(1))(p_(1),t_(1)^(2))(p_(2),t_(2)^(2))dots(p_(N),t_(N)^(2))dots(p_{1},t_{1}^{1})(p_{2},t_{2}^{1})\dots(p_{N},t_{N}^{1})(p_{1},t_{1}^{2})(p_{ 2},t_{2}^{2})\dots(p_{N},t_{N}^{2})\dots$(p_1,t_1^1)(p_2,t_2^1)\dots (p_N,t_N^1)(p_1,t_1^2)(p_2,t_2^2)\dots (p_N,t_N^2)\dots$, where $p_1,p_2,\dots ,p_N$$p_1,p_2,\dots ,p_N$p_(1),p_(2),dots,p_(N)p_{1},p_{2},\dots,p_{N}$p_1,p_2,\dots ,p_N$ is the repeated phase sequence and $t_i^j$$t_i^j$t_(i)^(j)t_{i}^{j}$t_i^j$ is the starting time of phase $p_i$$p_i$p_(i)p_{i}$p_i$ in the $j$$j$jj$j$-th cycle. Specifically, $C^j=t_1^{j+1}-t_1^j$$C^j=t_1^{j+1}-t_1^j$C^(j)=t_(1)^(j+1)-t_(1)^(j)C^{j}=t_{1}^{j+1}-t_{1}^{j}$C^j=t_1^{j+1}-t_1^j$ is the cycle length of the $j$$j$jj$j$-th phase cycle, and $\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}${(t_(2)^(j)-t_(1)^(j))/(C^(j)),dots,(t_(N)^(j)-t_(N-1)^(j))/(C^(j))}\{\frac{t_{2}^{j}-t_{1}^{j}}{C^{j}},\dots,\frac{t_{N}^{j}-t_{N-1}^{j}}{C^{j}}\}$\{\frac{t_2^j-t_1^j}{C^j},\dots ,\frac{t_N^j-t_{N-1}^j}{C^j}\}$ is the phase split of the $j$$j$jj$j$-th phase cycle. Existing traffic signal control methods usually repeats similar phase sequence throughout the day.

Objective

• Travel time. In traffic signal control, travel time of a vehicle is defined as the time different between the time one car enters the system and the time it leaves the system. One of the most common goals is to minimize the average travel time of vehicles in the network.
• Queue length. The queue length of the road network is the number of queuing vehicles in the road network.
• Number of stops. The number of stops of a vehicle is the total times that a vehicle experienced.
• Throughput. The throughput is the number of vehicles that have completed their trip in the road network during throughout a period.

Special Considerations

1. Yellow and all-red time. A yellow signal is usually set as a transition from a green signal to a red one. Following the yellow, there is an all-red period during which all the signals in an intersection are set to red. The yellow and all-red time, which can last from 3 to 6 seconds, allow vehicles to stop safely or pass the intersection before vehicles in conflicting traffic movements are given a green signal.
2. Minimum green time. Usually, a minimum green signal time is required to ensure pedestrians moving during a particular phase can safely pass through the intersection.
3. Left turn phase. Usually, a left-turn phase is added when the left-turn volume is above certain threshold.

2.2 Background of Reinforcement Learning

Single Agent Reinforcement learning

Problem setting

2.3 Literature

Agent formulation

2.3.1.1 Reward

2.3.1.2 State

2.3.1.3 Action scheme

Policy Learning

2.3.2.1 Value-based methods

Joint action learners

2.3.3.2 Independent learners

2.3.3.3 Sizes of Road network

2.4 Conclusion

Chapter 3 Basic Formulation for Traffic Signal Control

3.1 Introduction

3.2 Related work

Conventional Traffic Light Control

Reinforcement Learning for Traffic Light Control

3.3 Problem Definition

3.4 Method

Framework

Agent Design

• State. Our state is defined for one intersection. For each lane $\mathsf{i}$$\mathsf{i}$i\mathsf{i}$\mathsf{i}$ at this intersection, the state component includes queue length ${\mathsf{L}}_{\mathsf{i}}$${\mathsf{L}}_{\mathsf{i}}$L_(i)\mathsf{L}_{\mathsf{i}}${\mathsf{L}}_{\mathsf{i}}$, number of vehicles ${\mathsf{V}}_{\mathsf{i}}$${\mathsf{V}}_{\mathsf{i}}$V_(i)\mathsf{V}_{\mathsf{i}}${\mathsf{V}}_{\mathsf{i}}$, updated waiting time of vehicles ${\mathsf{W}}_{\mathsf{i}}$${\mathsf{W}}_{\mathsf{i}}$W_(i)\mathsf{W}_{\mathsf{i}}${\mathsf{W}}_{\mathsf{i}}$. In addition, the state includes an image representation of vehicles' position $\mathsf{M}$$\mathsf{M}$M\mathsf{M}$\mathsf{M}$, current phase ${\mathsf{P}}_c$${\mathsf{P}}_c$P_(c)\mathsf{P}_{c}${\mathsf{P}}_c$ and next phase ${\mathsf{P}}_n$${\mathsf{P}}_n$P_(n)\mathsf{P}_{n}${\mathsf{P}}_n$.
• Action. Action is defined as $\mathsf{a}=1$$\mathsf{a}=1$a=1\mathsf{a}=1$\mathsf{a}=1$: change the light to next phase ${\mathsf{P}}_n$${\mathsf{P}}_n$P_(n)\mathsf{P}_{n}${\mathsf{P}}_n$, and $\mathsf{a}=0$$\mathsf{a}=0$a=0\mathsf{a}=0$\mathsf{a}=0$: keep the current phase ${\mathsf{P}}_c$${\mathsf{P}}_c$P_(c)\mathsf{P}_{c}${\mathsf{P}}_c$.
• Reward. As is shown in Equation 3.3, reward is defined as a weighted sum of the following factors: 1. Sum of queue length $\mathsf{L}$$\mathsf{L}$L\mathsf{L}$\mathsf{L}$ over all approaching lanes, where $\mathsf{L}$$\mathsf{L}$L\mathsf{L}$\mathsf{L}$ is calculated as the total number of waiting vehicles on the given lane. A vehicle with a speed of less than 0.1 m/s is considered as waiting. 2. Sum of delay $\mathsf{D}$$\mathsf{D}$D\mathsf{D}$\mathsf{D}$ over all approaching lanes, where the delay ${\mathsf{D}}_{\mathsf{i}}$${\mathsf{D}}_{\mathsf{i}}$D_(i)\mathsf{D}_{\mathsf{i}}${\mathsf{D}}_{\mathsf{i}}$ for lane $\mathsf{i}$$\mathsf{i}$i\mathsf{i}$\mathsf{i}$ is defined in Equation 3.1, where the lane speed is the average speed of vehicles

3. Sum of updated waiting time $\mathsf{W}$$\mathsf{W}$W\mathsf{W}$\mathsf{W}$ over all approaching lanes. This equals to the sum of $\mathsf{W}$$\mathsf{W}$W\mathsf{W}$\mathsf{W}$ over all vehicles on approaching lanes. The updated waiting time $\mathsf{W}$$\mathsf{W}$W\mathsf{W}$\mathsf{W}$ for vehicle j at time $t$$t$tt$t$ is defined in Equation 3.2. Note that the updated waiting time of a vehicle is reset to 0 every time it moves. For example, if a vehicle's speed is 0.01m/s from 0s to 15s, 5m/s from 15s to 30s, and 0.01m/s from 30s to 60s, ${\mathsf{W}}_{\text{j}}$${\mathsf{W}}_{\text{j}}$W_("j")\mathsf{W}_{\text{j}}${\mathsf{W}}_{\text{j}}$ is 15 seconds, 0 seconds and 30 seconds when $t=$$t=$t=t=$t=$15s, 30s and 60s relatively. $${\mathsf{W}}_{\text{j}}(t)=\{\begin{array}{ll}{\mathsf{W}}_{\text{j}}(t-1)+1& vehiclespeed<0.1\\ 0& vehiclespeed\ge 0.1\end{array}$$$${\mathsf{W}}_{\text{j}}(t)=\left\{\begin{array}{ll}{\mathsf{W}}_{\text{j}}(t-1)+1& vehiclespeed<0.1\\ 0& vehiclespeed\ge 0.1\end{array}\right.$$W_("j")(t)={[W_("j")(t-1)+1,vehiclespeed < 0.1],[0,vehiclespeed >= 0.1]:}\mathsf{W}_{\text{j}}(t)=\begin{cases}\mathsf{W}_{\text{j}}(t-1)+1&vehicle\ speed<0.1\\ 0&vehicle\ speed\geq 0.1\end{cases}$${\mathsf{W}}_{\text{j}}(t)=\{\begin{array}{ll}{\mathsf{W}}_{\text{j}}(t-1)+1& vehiclespeed<0.1\\ 0& vehiclespeed\ge 0.1\end{array}$$ (3.2)
4. Indicator of light switches $\mathsf{C}$$\mathsf{C}$C\mathsf{C}$\mathsf{C}$, where $\mathsf{C}=0$$\mathsf{C}=0$C=0\mathsf{C}=0$\mathsf{C}=0$ for keeping the current phase, and $\mathsf{C}=1$$\mathsf{C}=1$C=1\mathsf{C}=1$\mathsf{C}=1$ for changing the current phase.
5. Total number of vehicles $\mathsf{N}$$\mathsf{N}$N\mathsf{N}$\mathsf{N}$ that passed the intersection during time interval $\mathrm{\Delta }t$$\mathrm{\Delta }t$Delta t\Delta t$\mathrm{\Delta }t$ after the last action a.
6. Total travel time of vehicles $\mathsf{T}$$\mathsf{T}$T\mathsf{T}$\mathsf{T}$ that passed the intersection during time interval $\mathrm{\Delta }t$$\mathrm{\Delta }t$Delta t\Delta t$\mathrm{\Delta }t$ after the last action a, defined as the total time (in minutes) that vehicles spent on approaching lanes. $$\begin{array}{r}Reward=w_1\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{L}}_{\text{i}}+w_2\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{D}}_{\text{i}}+w_3\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{W}}_{\text{i}}+\\ w_4\ast \mathsf{C}+w_5\ast \mathsf{N}+w_6\ast \mathsf{T}.\end{array}$$$$\begin{array}{r}Reward=w_1\ast \sum _{\text{i}\in \mathsf{I}} {\mathsf{L}}_{\text{i}}+w_2\ast \sum _{\text{i}\in \mathsf{I}} {\mathsf{D}}_{\text{i}}+w_3\ast \sum _{\text{i}\in \mathsf{I}} {\mathsf{W}}_{\text{i}}+\\ w_4\ast \mathsf{C}+w_5\ast \mathsf{N}+w_6\ast \mathsf{T}.\end{array}$${:[Reward=w_(1)**sum_("i"inI)L_("i")+w_(2)**sum_("i"inI)D_("i")+w_(3)**sum_("i"inI)W_("i")+],[w_(4)**C+w_(5)**N+w_(6)**T.]:}\begin{split} Reward=w_{1}*\sum_{\text{i}\in\mathsf{I}}\mathsf{ L}_{\text{i}}+w_{2}*\sum_{\text{i}\in\mathsf{I}}\mathsf{D}_{\text{i}}+w_{3}* \sum_{\text{i}\in\mathsf{I}}\mathsf{W}_{\text{i}}+\\ w_{4}*\mathsf{C}+w_{5}*\mathsf{N}+w_{6}*\mathsf{T}.\end{split}$$\begin{array}{r}Reward=w_1\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{L}}_{\text{i}}+w_2\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{D}}_{\text{i}}+w_3\ast \sum _{\text{i}\in \mathsf{I}}{\mathsf{W}}_{\text{i}}+\\ w_4\ast \mathsf{C}+w_5\ast \mathsf{N}+w_6\ast \mathsf{T}.\end{array}$$ (3.3)

Network Structure

Memory Palace and Model Updating

Experiment Setting

Parameter Setting

Evaluation Metric

• Reward: average reward over time. Defined in Equation 1, the reward is a combination of several terms (positive and negative terms), therefore, the range of reward is from $-\mathrm{\infty }$$-\mathrm{\infty }$-oo-\infty$-\mathrm{\infty }$ to $\mathrm{\infty }$$\mathrm{\infty }$oo\infty$\mathrm{\infty }$. Under specific configuration, there will be an upper bound for the reward when all cars move freely without any stop or delay.
• Queue length: average queue length over time, where the queue length at time $t$$t$tt$t$ is the sum of $\mathsf{L}$$\mathsf{L}$L\mathsf{L}$\mathsf{L}$ (defined in Section 3.4.2) over all approaching lanes. A smaller queue length means there are fewer waiting vehicles on all lanes.
• Delay: average delay over time, where the delay at time $t$$t$tt$t$ is the sum of $\mathsf{D}$$\mathsf{D}$D\mathsf{D}$\mathsf{D}$ (defined in Equation 3.1) of all approaching lanes. A lower delay means a higher speed of all lanes.
• Duration: average travel time vehicles spent on approaching lanes (in seconds). It is one of the most important measures that people care when they drive on the road. A smaller duration means vehicles spend less time passing through the intersection.

Compared Methods

• Self-Organizing Traffic Light Control (SOTL)[74]. This method controls the traffic light according to the current traffic state, including the eclipsed time and the number of vehicles waiting at the red light. Specifically, the traffic light will change when the number of waiting cars is above a hand-tuned threshold.
• Deep Reinforcement Learning for Traffic Light Control (DRL). Proposed in [10], this method applies DQN framework to select optimal light configurations for traffic intersections. Specifically, it solely relies on the original traffic information as an image.

• IntelliLight (Base). Using the same network structure and reward function defined as in Section 3.4.2 and 3.4.3. This method is without Memory Palace and Phase Gate.
• IntelliLight (Base+MP). By adding Memory Palace in psychology to IntelliLightBase, we store the samples from different phase and time in separate memories.
• IntelliLight (Base+MP+PG). This is the model adding two techniques (Memory Palace and Phase Gate).

Datasets

Synthetic data

Real-world data

Performance on Synthetic Data

3.5.6.1 Comparison with state-of-the-art methods

3.5.6.2 Comparison with variants of our proposed method

3.5.6.3 Interpretation of learned signal

Performance on Real-world Data

Comparison of different methods

3.5.7.2 Observations with respect to real traffic

3.6 Conclusion

Chapter 4 Formulating the Learning Objective

4.1 Introduction

4.2 Related Work

4.3 Preliminaries and Notations