CRII: CSR: RUI: Novel Approaches for Task Graph Scheduling Algorithm Design, Evaluation, and Comparison

The task scheduling problem is fundamental to distributed computing and has thus been studied extensively over the past several decades. Better task scheduling approaches will improve how computational tasks execute on distributed computing systems, which support a vast array of important domains including scientific workflows, machine learning pipelines, and Internet of Things (IoT) applications. Efficient task scheduling helps these systems run faster, use resources more effectively, and deliver more reliable performance. Although many scheduling algorithms exist, it is not well understood how they perform across different types of real-world scenarios, or in which kinds of scenarios they may perform poorly. This work builds upon an open-source framework to design, evaluate, and compare scheduling algorithms under realistic conditions to better understand their capabilities. With this resulting insight, the work explores new task scheduling approaches that leverage the strengths of prior approaches and employs machine learning to enable intelligent scheduling. The project provides hands-on undergraduate research opportunities at a primarily undergraduate institution, strengthening the pipeline of future computing researchers and helping address national workforce development needs. These efforts support national research infrastructure, scientific progress, and the development of robust, efficient, and scalable computing systems for the public good.

This project extends an open-source framework to integrate with widely used distributed computing platforms such as Pegasus, Parsl, and Docker, enabling empirical evaluation of scheduling algorithms on real applications and real systems. This work also adds support for constraint-based scheduling and scheduling in stochastic environments (both of which more accurately reflect the uncertainties and constraints common in practice). The project introduces methods for automatically designing hybrid scheduling algorithms by identifying and combining the strengths of existing ones using adversarial analysis. It also explores how adversarially generated problem instances can be used to improve the training of machine learning-based schedulers by enriching their training data with informative, diverse examples. By providing standardized tools and methodologies, this work promotes reproducibility, encourages innovation, and empowers a broad range of researchers to better understand and improve task scheduling in heterogeneous distributed environments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Award Number: 2451267
Principal Investigator: Jared Coleman
Funds Obligated: $150,000
State: CA