OOCL-DDQN: Online Evaluation and Offline Training-Based Clipped Double DQN for Automated Anesthesia Control

Anesthesia control is critical in surgical procedures. Nowadays, various methods have been proposed for automated anesthesia control. However, traditional control model-based methods and PK/PD-based methods cannot fully capture the individual characteristics of patients. Machine learning-based methods are not suitable for dealing with large-scale clinical anaesthesia data. The existing deep reinforcement learning-based (DRL) methods cannot provide both stable and reliable anesthesia strategies while ensuring the agent's adaptability to environmental changes. To address these issues, in this paper, we propose a deep reinforcement learning model named OOCL-DDQN for automated anesthesia control. Specifically, we propose an online evaluation and offline training mechanism to well avoid excessive reliance on the environment models. To enhance the model stability, we design a clip method to optimize the Bellman equation in Double DQN. Besides, we propose a fixed-order random sampling method to enable the agent to effectively learn the relationship between anesthesia state and anesthetic dosage in different patients. In data preprocessing, we design a same-frequency resampling method to ensures that the resampled data obey the Markov property. To evaluate the performance of our proposed method, we conduct extensive experiments on a real-world dataset. The experiment results show that our proposed model gets better performances than the other state-of-the-art methods.