Optimizing job scheduling by using broad learning to predict execution times on HPC clusters

Small and middle size high-performance computing clusters are very popular for various applications. How to utilize the accumulated log data generated in the past to optimize job scheduling using machine learning techniques is an interesting problem. Most of the current work use the common machine learning algorithms, such as the multivariate linear regression and polynomial model, to predict job runtime and optimize job scheduling. They either ignore the interference among job features or require a high time overhead for improving the prediction accuracy. In this paper, we propose to implement and improve broad learning algorithm for predicting the execution times of new coming jobs more accurately and efficiently. The experimental results showed that the proposed method can obtain high prediction accuracy with a negligible time overhead. And the predicted job execution time can help improve the efficiency of job scheduling and HPC systems.