Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Compute Node Failure Recovery Policy

This section describes the recovery policy for handling failures in compute nodes.

Compute Node Failures

A failure is detected when the compute node is deemed to be operating abnormally. The following are typical indicators of such abnormal operation:

• Some devices installed on the compute node are malfunctioning
• Communication with the compute node cannot be established

Compute Node Failure Recovery Policy

When a failure is confirmed in a compute node, the following recovery procedures will be implemented according to the established policy:

• Without prior notification to users, the workloads (Kubernetes Pods) running on the affected compute node will be terminated, and the compute node will be rebooted.
  • The terminated workloads will be automatically recreated on either the same compute node or another available node, following the rescheduling policy.
    • Bare Pods (those directly created by users without using higher-level resources like Deployments or Jobs) will not be automatically recreated.
    • We strongly recommend avoiding the direct creation of Pod resources.
• For dedicated nodes where rebooting does not resolve the issue, replacement with an alternative compute node will be performed.

Recommended Preparations for Compute Node Failures

• Always store critical data such as computation results and outputs in persistent storage.
• For stateful workloads, thoroughly plan and prepare procedures for data persistence and recovery.

Important

Workloads and processes may terminate for reasons beyond maintenance or failures. Always store critical data in persistent storage.