Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Creating Distributed Batch Jobs

This page explains how to create ParallelJobs for executing distributed batch processing on a PFCP cluster.

Note

As this is an experimental feature, the API schema may change in the future.

Overview

A ParallelJob is a Kubernetes custom resource designed to easily deploy and manage distributed batch jobs that operate in coordination across multiple nodes. ParallelJobs offer the following key features:

  • Support for multiple frameworks: Supports distributed processing frameworks such as MPI.
  • Gang Scheduling: Ensures all Pods within a distributed batch job are scheduled simultaneously.
  • Retry on job failure: Automatically reruns the entire ParallelJob if any single job fails.

Runtimes Supported by ParallelJob

Note

While we plan to support multiple runtimes in the future, currently only the Open MPI runtime is available.

Open MPI

Supports parallel computing using the Open MPI implementation of the Message Passing Interface (MPI). For Open MPI jobs, a single launcher Pod connects to multiple worker Pods via SSH to perform parallel computation using MPI. The launcher Pod also serves as a worker with rank=0.

Below is an example configuration for a ParallelJob using the MPI runtime:

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  # Specifies the MPI runtime to use
  runtimeRef:
    name: mpi-openmpi

  # Number of Pods including both launcher and worker Pods.
  # For MPI runtimes, there is one launcher Pod and the remainder are worker Pods.
  numPods: 3
  # Number of MPI processes per Pod, which corresponds to the slots in the MPI hostfile.
  # For accelerator-based computations, this typically specifies the number of accelerators per Pod, for flat MPI it specifies the number of CPU cores, and for hybrid parallelism, you would typically specify 1.
  numProcPerPod: 2

  # Configuration for the Pod template for the launcher Pod. Specified in PodTemplateSpec format.
  # If the `main` container is not included, the ParallelJob creation will fail.
  launcher:
    spec:
      containers:
      - name: main # The container running MPI should be named `main`.
        image: ghcr.io/pfnet/parallel-controller/openmpi:v0.1.0
        command:
        - sh
        - -c
        - |
          mpirun --allow-run-as-root sh -c '
            cat > hello_mpi.c << "EOF"
            #include <mpi.h>
            #include <stdio.h>

            int main(int argc, char *argv[]) {
              MPI_Init(&argc, &argv);
              int rank, world_size;
              MPI_Comm_rank(MPI_COMM_WORLD, &rank);
              MPI_Comm_size(MPI_COMM_WORLD, &world_size);
              printf("Hello from MPI process %d rank in %d processes\n", rank, world_size);
              MPI_Finalize();
              return 0;
            }
            EOF
            mpicc -o hello_mpi hello_mpi.c
            ./hello_mpi'

  # (Optional) Configuration for the Pod template for worker Pods. Specified in PodTemplateSpec format.
  # Worker settings inherit from the launcher and automatically generate a container named `main` for communication with the launcher.
  # worker: nil

Note

Using Custom Container Images

When using custom container images with the Open MPI runtime, the following software must be included:

  • Open MPI
  • ssh (SSH client)
  • sshd (SSH server)

Example Configurations for Different Scenarios

Below are example configurations commonly used in actual deployment scenarios.

Retry Settings for Job Failures

By default, if any job within a ParallelJob fails, the entire ParallelJob will be retried up to a maximum of 3 times. To change the retry count, set the .spec.failurePolicy.maxRestarts field. Below is an example setting the retry count to 1.

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  failurePolicy:
    # Maximum number of times to rerun the entire ParallelJob when a job fails. (Default: 3)
    maxRestarts: 1
...

Note

Job interruptions due to preemption or eviction are not counted towards the retry limit.

Specifying GPU/RDMA Resources

When using GPU or RDMA resources in distributed jobs, specify the corresponding resource requirements in the Pod template. NCCL and UCX configurations will be automatically generated and set as environment variables in the container. Below is an example requesting GPU and RDMA devices for the launcher Pod.

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: sample-job
spec:
  launcher:
    spec:
      containers:
      - name: main
        command: ["mpirun", "--allow-run-as-root", "something-using-gpu-rdma"]
        resources:
          limits:
            nvidia.com/gpu: "2"
            preferred.jp/rdma: "1"
...

Note

RDMA resources are only available on the following clusters:

  • IK1-01
  • YH1-01

Troubleshooting

If issues occur during distributed job execution, follow these steps to troubleshoot.

Checking Job Status

To check the status of a ParallelJob, execute the following command:

# Check ParallelJob status
kubectl get paralleljobs sample-job

# View detailed information including events
kubectl describe paralleljobs sample-job

To check the status of related Jobs/Pods, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Check the status of related Jobs/Pods
kubectl get jobs,pods -l jobset.sigs.k8s.io/jobset-name=${jobset_name}

Checking Pod Logs

To retrieve logs from the rank=0 Pod, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Display logs
kubectl logs -f $(kubectl get pod -l jobset.sigs.k8s.io/jobset-name=${jobset_name},jobset.sigs.k8s.io/job-global-index=0 -o name)

Note

Logs can only be checked if the Pod exists. For successful completion, all Pods remain, while for failed completion only the failed Pod remains. Additionally, deleting a ParallelJob will also remove all associated Pods.