Prediction of job characteristics for intelligent resource allocation in HPC systems: a survey and future directions

Zhengxiong Hou, Hong Shen, Xingshe Zhou, Jianhua Gu, Yunlan Wang, Tianhai Zhao

Research output: Contribution to journalReview articlepeer-review

5 Citations (Scopus)

Abstract

Nowadays, high-performance computing (HPC) clusters are increasingly popular. Large volumes of job logs recording many years of operation traces have been accumulated. In the same time, the HPC cloud makes it possible to access HPC services remotely. For executing applications, both HPC end-users and cloud users need to request specific resources for different workloads by themselves. As users are usually not familiar with the hardware details and software layers, as well as the performance behavior of the underlying HPC systems. It is hard for them to select optimal resource configurations in terms of performance, cost, and energy efficiency. Hence, how to provide on-demand services with intelligent resource allocation is a critical issue in the HPC community. Prediction of job characteristics plays a key role for intelligent resource allocation. This paper presents a survey of the existing work and future directions for prediction of job characteristics for intelligent resource allocation in HPC systems. We first review the existing techniques in obtaining performance and energy consumption data of jobs. Then we survey the techniques for single-objective oriented predictions on runtime, queue time, power and energy consumption, cost and optimal resource configuration for input jobs, as well as multi-objective oriented predictions. We conclude after discussing future trends, research challenges and possible solutions towards intelligent resource allocation in HPC systems.

Original languageEnglish
Article number165107
JournalFrontiers of Computer Science
Volume16
Issue number5
DOIs
Publication statusPublished - Oct 2022
Externally publishedYes

Keywords

  • cloud computing
  • high-performance computing
  • intelligent resource allocation
  • job characteristics
  • machine learning
  • performance prediction

Fingerprint

Dive into the research topics of 'Prediction of job characteristics for intelligent resource allocation in HPC systems: a survey and future directions'. Together they form a unique fingerprint.

Cite this