Optimizing job scheduling by using broad learning to predict execution times on HPC clusters

Zhengxiong Hou, Hong Shen, Qiying Feng, Zhiqi Lv, Junwei Jin, Xingshe Zhou, Jianhua Gu

Research output: Contribution to journalArticlepeer-review

Abstract

Small and middle size high-performance computing clusters are very popular for various applications. How to utilize the accumulated log data generated in the past to optimize job scheduling using machine learning techniques is an interesting problem. Most of the current work use the common machine learning algorithms, such as the multivariate linear regression and polynomial model, to predict job runtime and optimize job scheduling. They either ignore the interference among job features or require a high time overhead for improving the prediction accuracy. In this paper, we propose to implement and improve broad learning algorithm for predicting the execution times of new coming jobs more accurately and efficiently. The experimental results showed that the proposed method can obtain high prediction accuracy with a negligible time overhead. And the predicted job execution time can help improve the efficiency of job scheduling and HPC systems.

Original languageEnglish
JournalCCF Transactions on High Performance Computing
DOIs
Publication statusAccepted/In press - 2023
Externally publishedYes

Keywords

  • Broad learning
  • HPC clusters
  • Job scheduling
  • Parallel systems
  • Runtime prediction

Fingerprint

Dive into the research topics of 'Optimizing job scheduling by using broad learning to predict execution times on HPC clusters'. Together they form a unique fingerprint.

Cite this