Proactive resource management to optimize distributed workflow executions
Key: WSLS23
Author: Joel Witzke, Florian Schintke, Ansgar Lößer, Björn Scheuermann
Date: December 2023
Kind: In proceedings
Publisher: IEEE
Abstract: Scientific workflows have received increasing interest and are used in many scientific fields to gather, analyze, and process significant amounts of data. However, their tasks are usually treated as black boxes, and their behavior remains unconsidered for resource allocations, which can lead to subpar resource allocations with typical scheduling. Although not done yet, it should be possible to observe such tasks, learn their behavior, and use this knowledge to improve future executions. As workflows and their tasks are often executed multiple times on a massive scale, even a slight improvement per execution may save hours of execution time and significant amounts of energy.To achieve this goal, we develop an innovative approach to model task executions and predict resource usage. The prediction is embedded in a feedback loop to repeatedly improve the models and to closely track workflow executions to make predictions and resource allocations accurate.
Official URL

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, not withstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.