One post tagged with "ml-training"

Batch and ML workloads often need resources that aren't known at Job creation time. The optimal allocation depends on cluster capacity, queue priorities, and hardware availability — all of which can change between submission and execution.

Kubernetes v1.36 promotes a feature to beta that solves this: mutable pod resources for suspended Jobs.

Before this, resource requirements in a Job's pod template were immutable. If a queue controller determined that a suspended Job should run with different resources, the only option was to delete and recreate the Job — losing metadata, status, and history.