Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Creating a Distributed Batch Job

This page explains how to create a ParallelJob for executing distributed batch processing on a PFCP cluster.

Note

As this is an experimental feature, the API schema may change in the future.

Overview

A ParallelJob is a Kubernetes custom resource designed to simplify the deployment and management of distributed batch jobs that coordinate across multiple nodes. ParallelJobs offer the following key features:

  • Support for multiple frameworks: Supports distributed processing frameworks such as MPI.
  • Gang Scheduling: Ensures all Pods within a distributed batch job are scheduled simultaneously.
  • Job failure retries: Automatically retries the entire ParallelJob if any single job fails.

Runtimes Supported by ParallelJob

Note

While we plan to support multiple runtimes in the future, currently only the Open MPI runtime is available.

Open MPI

Supports parallel computing using the Open MPI implementation of the Message Passing Interface (MPI). In an Open MPI job, a single launcher Pod connects to multiple worker Pods via SSH to execute parallel computations using MPI. The launcher Pod also functions as a worker with rank=0.

Below is an example configuration for a ParallelJob using the MPI runtime:

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  # Specifies the MPI runtime to use
  runtimeRef:
    name: mpi-openmpi

  # Number of Pods including both launcher and worker Pods.
  # For MPI runtimes, there is one launcher Pod and the remainder are worker Pods.
  numPods: 3
  # Number of MPI processes per Pod, which corresponds to the "slots" in the MPI hostfile.
  # For accelerator-based computations, this should specify the number of accelerators per Pod; for flat MPI, the number of CPU cores; and for hybrid parallelism, typically set to 1.
  numProcPerPod: 2

  # Configuration for the launcher Pod template. Specified in PodTemplateSpec format.
  # If the `main` container is missing, the ParallelJob creation will fail.
  launcher:
    spec:
      containers:
      - name: main # The container running MPI should be named `main`.
        image: ghcr.io/pfnet/parallel-controller/openmpi:v0.1.0
        command:
        - sh
        - -c
        - |
          mpirun --allow-run-as-root sh -c '
            cat > hello_mpi.c << "EOF"
            #include <mpi.h>
            #include <stdio.h>

            int main(int argc, char *argv[]) {
              MPI_Init(&argc, &argv);
              int rank, world_size;
              MPI_Comm_rank(MPI_COMM_WORLD, &rank);
              MPI_Comm_size(MPI_COMM_WORLD, &world_size);
              printf("Hello from MPI process %d rank in %d processes\n", rank, world_size);
              MPI_Finalize();
              return 0;
            }
            EOF
            mpicc -o hello_mpi hello_mpi.c
            ./hello_mpi'

  # (Optional) Configuration for the worker Pod template. Specified in PodTemplateSpec format.
  # Worker settings inherit from the launcher, and an automatically generated container for communication with the launcher will be named `main`.
  # worker: nil

Note

Using custom container images

When using custom container images with the Open MPI runtime, the following software must be included:

  • Open MPI
  • ssh (SSH client)
  • sshd (SSH server)

Example Configurations for Different Scenarios

Below are example configurations commonly used in practical deployment scenarios.

Retry Configuration for Job Failures

By default, if any job within a ParallelJob fails, the entire ParallelJob will be retried up to a maximum of 3 times. To change the retry count, set the .spec.failurePolicy.maxRestarts field. Below is an example setting the retry count to 1.

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  failurePolicy:
    # Maximum number of times to rerun the entire ParallelJob on job failure. (Default: 3)
    maxRestarts: 1
...

Note

Job interruptions caused by preemption or eviction are not counted towards the retry limit.

Specifying GPU/RDMA Resources

When using GPU or RDMA resources in a distributed job, specify the corresponding resource requirements in the Pod template. While NCCL and UCX configuration files are automatically generated, they must be explicitly sourced. Below is an example requesting GPU and RDMA devices for the launcher Pod and loading NCCL and UCX configuration files (this will be automated in future updates).

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: sample-job
spec:
  launcher:
    spec:
      containers:
      - name: main
        command: ["/bin/sh", "-c"]
        args:
        # Note: While NCCL and UCX configuration files are automatically generated, they must be explicitly sourced.
        - |
          [ -f "$RDMA_NCCL_CONF" ] && . "$RDMA_NCCL_CONF"
          [ -f "$RDMA_UCX_CONF" ] && . "$RDMA_UCX_CONF"

          mpirun --allow-run-as-root something-using-gpu-rdma
        resources:
          limits:
            nvidia.com/gpu: "2"
            preferred.jp/rdma: "1"
...

Note

RDMA resources are only available on the following clusters:

  • IK1-01
  • YH1-01

Troubleshooting

When encountering issues during distributed job execution, follow these steps to troubleshoot:

Checking Job Status

To check the status of a ParallelJob, execute the following command:

# Check ParallelJob status
kubectl get paralleljobs sample-job

# View detailed information including events
kubectl describe paralleljobs sample-job

To check the status of related Job/Pods, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Check the status of related Jobs/Pods
kubectl get jobs,pods -l jobset.sigs.k8s.io/jobset-name=${jobset_name}

Checking Pod Logs

To retrieve logs from the rank=0 Pod, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Display logs
kubectl logs -f $(kubectl get pod -l jobset.sigs.k8s.io/jobset-name=${jobset_name},jobset.sigs.k8s.io/job-global-index=0 -o name)

Note

Logs can only be checked if the Pod exists. For successful completion, all Pods remain; for error termination, only the errored Pod remains. Additionally, deleting a ParallelJob will also remove all associated Pods.