Distributed multi-agent reinforcement learning with neighborhood-aware spatio-temporal coordination for traffic network signal control

To address the challenge of traffic congestion in urban areas, researchers have developed a novel distributed multi-agent reinforcement learning (MADRL) framework called SANIS-MARL. This innovative approach tackles the limitations of previous methods by incorporating a spatio-temporal attention network and neighborhood information sharing.

The key features of SANIS-MARL include a multi-head attention mechanism combined with Long Short-Term Memory (LSTM) networks to capture temporal dependencies in traffic patterns across various time scales. Spatially, a Graph Attention Network v2 (GATv2) dynamically learns the relative significance of neighboring intersections based on real-time traffic conditions.

Additionally, a neighborhood information-sharing mechanism is employed, where intersections are grouped based on geographic proximity, and the influence of neighboring intersections is adaptively weighted through spatial attention.

By coordinating signals locally and mitigating non-stationarity issues typically faced in MADRL, SANIS-MARL demonstrates superior performance compared to conventional approaches and state-of-the-art multi-agent controllers. Experiments conducted on an urban network scenario show significant improvements in reducing queue lengths, waiting times, travel times, and emissions.

Furthermore, attention heatmaps generated by the proposed method provide visual evidence of its ability to detect early warning signs of congestion waves, offering an interpretable and scalable solution for cooperative traffic signal control.

This groundbreaking work was made possible through the support of research grants from the National Natural Science Foundation of China (62203021, 52402393, 62503023) and the China Postdoctoral Science Foundation (2023M730150). The research was conducted by Yongnan Zhang, Sirui Peng, Xiaobin Luo, Xin Gu, Xiaolu Jia, and Yunxuan Li at the College of Metropolitan Transportation, Beijing University of Technology, Beijing, China.