Sequential State Q-learning Uplink Resource Allocation in Multi-AP 802.11be Network

Expected high demand of user applications in the WLAN is a driver for WLANs to share radio resources more efficiently. The move to 802.11be with OFDMA and MU-MIMO makes Radio Resource Management (RRM) a multi-dimensional problem in a complex wireless environment. Traditionally, the way that an RRM problem is formulated always leads to either a large state space or action space, which makes reinforcement learning impossible to be applied. In this paper, we propose a Sequential State Q-learning algorithm (SSQL) aimed at solving the Resource Unit (RU) allocation for scheduled uplink transmission to maximize system bitrate in a multi-AP 802.11be OFDMA network. The AP acts as the agent with the serving stations as 'states' and their RU allocations as 'actions'. The AP observes the wireless environment, continuously refreshing the Q-values of the state-action pairs and outputs the RU allocation to optimize the objective. Through simulations, we demonstrated that the performance of SSQL is 89.67% of the global optimal with very fast convergence, which makes it more practical for use in varying wireless networks.