Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Overview of Preferred Computing Platform (PFCP)

Preferred Computing Platform™ (PFCP™) is a cloud service developed by Preferred Networks specifically for deep learning and AI workloads. It provides exclusive access to the company’s proprietary accelerator series, the MN-Core™, enabling high-performance AI computing and exceptional computational efficiency.

Powerful Computational Boards and High-Speed Networking

Users can exclusively utilize multiple MN-Server 2 V1 servers, each equipped with eight MN-Core 2 boards. All nodes are interconnected via high-speed networking optimized for deep learning applications.

Fully Managed Service

Access a multi-tenant Kubernetes cluster ¹ that has been enhanced for deep learning and AI workloads. This platform supports a wide range of machine learning tasks, from large-scale distributed training to high-availability inference server operations. Additionally, it includes a managed monitoring service ² for monitoring workload performance.

Other Key Features

• Flexible management of user permissions and network access control lists (ACLs)
• Secure data storage using high-reliability persistent storage
• Pre-installed with numerous Kubernetes custom add-ons, allowing immediate deployment and monitoring of AI workloads
• Supports execution from any workload configuration file using PFCP-provided container images
  • Users can also bring their own managed container images
• Supports integration with public clouds, enabling secure access to public cloud resources without requiring high-privilege keys

1. Kubernetes is an open-source container orchestration platform hosted by the Cloud Native Computing Foundation (CNCF). It serves as an automated tool for deploying, scaling, and managing the availability of containerized applications and has become widely adopted as the standard in this field. ↩

2. The monitoring service consists of open-source tools including Grafana, Prometheus, and Alertmanager. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Glossary

This document explains the terminology used in Preferred Computing Platform (PFCP), a cloud service designed for AI/ML workloads.

Organizations

These are the fundamental resources required for using PFCP, including Kubernetes cluster access, portal usage, and authentication/authorization mechanisms. They are equivalent to what is generally referred to as “tenants” ¹.

To use PFCP, you must be invited by the organization you wish to access. When interacting with PFCP resources such as Kubernetes clusters or portals, you must select the appropriate organization.

Within a single organization, multiple PFCP clusters can be utilized, and multiple Namespaces can be created. Each Namespace supports both permission isolation and network segmentation. Therefore, there is no need to strictly separate organizations by region or usage purpose.

Root Namespaces and Sub-Namespaces

In PFCP, each organization is assigned a single Namespace, with additional Namespaces created as child Namespaces under this primary Namespace. The primary Namespace is referred to as a “root Namespace,” while the child Namespaces are called “sub-Namespaces.” For detailed information about PFCP Namespaces, please refer to Logical Cluster Division Using Namespaces.

Root Namespaces and sub-Namespaces are unique PFCP-specific terms that do not exist in standard Kubernetes clusters. Since they differ in specifications and handling, these specific terms are used when distinction is necessary. However, when describing general Kubernetes Namespace resources where distinction between root and sub-Namespaces is not required, the term Namespace is used instead.

User Roles (Organization Administrator vs. Regular User)

PFCP defines two distinct user roles: Organization Administrator and Regular User. For detailed information about these roles and the operations they permit, please refer to PFCP User Roles.

Meanwhile, Kubernetes provides Role/ClusterRole resources to implement Role-Based Access Control (RBAC) ². While PFCP also provides standard ClusterRoles ³, these ClusterRoles do not map one-to-one with PFCP user roles ⁴. Organization Administrators can create RoleBindings to customize mappings on a Namespace-by-Namespace basis.

Individual RoleBindings can be created for specific users, or users can be grouped into roles and RoleBindings created for these groups.

For detailed information about RoleBindings, please refer to Managing User Cluster Access Permissions. For comprehensive user management information, please refer to Managing Organization Users.

Monitoring Services

This term collectively refers to the managed services provided by PFCP: Grafana, Prometheus, and Alertmanager.

Identity-Aware Proxy (IAP)

This is a PFCP-provided mechanism for securely exposing workloads to the internet. Automatic authentication is configured during access, enabling secure deployment of workloads.

There are two variants based on the authentication method: API Identity-Aware Proxy (API IAP) and WebApp Identity-Aware Proxy (WebApp IAP). For detailed usage instructions, please refer to Exposing Workloads as Web APIs or Exposing Workloads as Web Applications.

1. This concept is analogous to Google Cloud projects or Amazon Web Service accounts. ↩

2. Role-Based Access Control ↩

3. Three roles are available: org-admin, org-edit, and org-view. ↩

4. Only Organization Administrators can be mapped to org-admin for both root Namespaces and all sub-Namespaces. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

PFCP Release Notes

July 2026

• English version of the PFCP User Guide is now available (Announcement)

June 2026

• Support for relational database execution via MOCO has been added (Documentation)

May 2026

• Support for automatic horizontal scaling of jobs via KEDA has been added (Documentation)
• Kubernetes: Upgrade to Kubernetes v1.35 has been completed across all clusters at our data centers (Documentation)

April 2026

• Kubernetes custom resource “SealedSecret” is now available for managing sensitive data using GitOps methodology (Documentation)

February 2026

• Kubernetes: “Headlamp” dashboard now available for viewing and managing resources directly in your browser (Documentation)
• PFCP: Release of MN-Core SDK v0.4 (Announcement)
• Kubernetes: Organizational resource quotas can now be viewed in Grafana dashboards (Announcement)
• PFCP: English version of the PFCP portal is now available (Announcement)
• Kubernetes: Updated ingress-nginx version to v1.14.3 to address security vulnerabilities (Announcement)
• Kubernetes: Enabled execution of GitHub Actions jobs (Documentation)

January 2026

• Kubernetes: Upgrade to Kubernetes v1.34 has been completed across all clusters at our data centers (Documentation)

November 2025

• PFCP: Ability to restrict Ingress visibility to selected users within your organization has been added (Announcement)
• Kubernetes: ParallelJob custom resource now available for executing distributed batch processing tasks (Documentation)
• Kubernetes: Filesystems are now available for persistent storage on shared nodes
• Kubernetes: Updated runc version to v1.3.3 to address security vulnerabilities (Announcement)

October 2025

• PFCP: Connection capability to workspaces via your locally installed Visual Studio Code is now available (Documentation)

September 2025

• Kubernetes: Support for automatic horizontal scaling of workloads via external event-driven triggers using KEDA has been added (Documentation)
• Kubernetes: Upgrade to Kubernetes v1.33 has been completed across all clusters at our data centers (Documentation)

August 2025

• Kubernetes: Snapshot functionality for persistent storage is now available (Documentation)

July 2025

• Issue resolved where settings created via AlertmanagerConfig custom resource were not being applied to Alertmanager
• PFCP: “Preset” feature now available, allowing users to pre-configure and share workspace settings (Documentation)
• PFCP: “API Identity-Aware Proxy” feature now available, enabling workloads to be exposed as web APIs (Documentation)

June 2025

• Kubernetes: Upgrade to Kubernetes v1.32 has been completed across all clusters at our data centers (Documentation)

May 2025

• Kubernetes: Two new PriorityClass options added for shared nodes: shared-standard and shared-best-effort (Documentation)
• PFCP: Users can now select Pod PriorityClass when creating workspaces through the PFCP portal

April 2025

• MN-Core: MN-Core SDK v0.2 has been released (for update details, see /opt/pfn/pfcomp/RELEASE_NOTES.md in the container image)
• Kubernetes: “Shared Nodes” now available as a new compute node type (Documentation)
• Kubernetes: “Workspace Suspend” feature now available, allowing computation resources to be conserved while preserving PersistentVolumes within the workspace (Documentation)

March 2025

• PFCP: PFCP portal now displays a list of available PFCP-provided container images (Documentation)

February 2025

• Kubernetes: “Workspace” feature now available, providing an interactive work environment accessible via browser (Documentation)
• Kubernetes: Upgrade to Kubernetes v1.31 has been completed across all clusters at our data centers (Documentation)
• Kubernetes: Organization administrators now able to manage Kubernetes Role resources (Documentation)

December 2024

• Kubernetes: PFCP portal now displays a list of dedicated nodes available to the organization
• PFCP: “User Groups” feature now available, allowing organization members to be organized into groups (Documentation)

November 2024

• Kubernetes: GitOps tool Flux now available across all clusters at our data centers (Documentation)

October 2024

• Kubernetes: File storage can now be shared across Namespaces within the same organization for persistent storage on SR1-01 clusters (Documentation)

September 2024

• Kubernetes: Upgrade to Kubernetes v1.30 has been completed across all clusters at our data centers (Documentation)

June 2024

• Kubernetes: Persistent storage now available on SR1-01 clusters (Documentation: Using File Storage, Using Block Storage)
• Kubernetes: The kubectl get reservednode command now provides information about proprietary nodes available to your organization (Documentation)
• Kubernetes: Upgrade to Kubernetes v1.29 has been completed across all clusters at our data centers (Documentation)

April 2024

• PFCP: Documentation site is now available
• PFCP: Portal site is now available

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Maintenance Policy

This section outlines the policies for scheduled and emergency maintenance activities.

Scheduled Maintenance Policy

PFCP implements regular scheduled maintenance to maintain system reliability.

Kubernetes Version Upgrade Policy

We regularly perform minor version upgrades to improve Kubernetes cluster stability, performance, and security.

Impact of Upgrades and Other Maintenance Activities on Clusters and Workloads

This section describes the user impacts of maintenance activities that involve temporary shutdowns of compute nodes, such as cluster upgrades.

Impact on Clusters and Compute Nodes

• Temporary instability in cluster operations may occur
• User-allocated dedicated nodes will have their upgraded compute nodes temporarily unavailable
  • For contracts with fewer than 4 dedicated nodes, one node will be temporarily unavailable; for contracts with 4+ nodes, up to 25% (rounded down) will be unavailable
  • For contracts with only one dedicated node, no nodes will be temporarily available for use
• Shared nodes will be temporarily unavailable by 25% (rounded down)

Impact on Workloads

• Workloads currently running on the compute nodes undergoing upgrade will be terminated
• Terminated workloads will be automatically recreated on different compute nodes or the same node, depending on rescheduling policies
  • Workloads directly created by users (Bare Pods) without using higher-level resources like Deployments or Jobs will not be automatically recreated
  • We strongly recommend avoiding direct creation of Pod resources

Warning

Important Note on Data Preservation

• Be aware that any data retained by workloads that is not stored in persistent storage will be lost
  • This includes data stored in local memory, temporary filesystems, etc.
• Data stored in persistent storage will be preserved

Recommended Preparations for Compute Node Maintenance

• Always store critical data such as computation results and outputs in persistent storage
• If your workloads are stateful, thoroughly plan and prepare procedures for data persistence and recovery

Important

Workloads and processes may stop for reasons beyond maintenance or failures. Always store critical data in persistent storage.

Emergency Maintenance Policy

Unscheduled maintenance may be conducted under circumstances where service continuity is difficult to maintain or when critical security vulnerabilities are discovered. When maintenance affects clusters, compute nodes, or workloads, we will provide advance notice of the planned activities.

Staying Informed About Maintenance Activities

All new maintenance information will be posted on the PFCP News Site. For details, please refer to Notification of New Features and Maintenance.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Compute Node Failure Recovery Policy

This section describes the recovery policy for handling failures in compute nodes.

Compute Node Failures

A failure is detected when the compute node is deemed to be operating abnormally. The following are typical indicators of such abnormal operation:

• Some devices installed on the compute node are malfunctioning
• Communication with the compute node cannot be established

Compute Node Failure Recovery Policy

When a failure is confirmed in a compute node, the following recovery procedures will be implemented according to the established policy:

• Without prior notification to users, the workloads (Kubernetes Pods) running on the affected compute node will be terminated, and the compute node will be rebooted.
  • The terminated workloads will be automatically recreated on either the same compute node or another available node, following the rescheduling policy.
    • Bare Pods (those directly created by users without using higher-level resources like Deployments or Jobs) will not be automatically recreated.
    • We strongly recommend avoiding the direct creation of Pod resources.
• For dedicated nodes where rebooting does not resolve the issue, replacement with an alternative compute node will be performed.

Recommended Preparations for Compute Node Failures

• Always store critical data such as computation results and outputs in persistent storage.
• For stateful workloads, thoroughly plan and prepare procedures for data persistence and recovery.

Important

Workloads and processes may terminate for reasons beyond maintenance or failures. Always store critical data in persistent storage.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Stay Updated on New Features and Maintenance Information

This section explains how to access the latest updates regarding new features and system maintenance for PFCP (Preferred Computing Platform), a cloud service designed for AI and ML workloads.

News Website

We maintain an up-to-date news website with all relevant information.

• https://pfcomputing.com/news/

Subscribe to New Updates Using an RSS Reader

Our news website publishes RSS feeds. You can subscribe to receive updates directly through your RSS reader.

• All categories: https://pfcomputing.com/news/index.xml
• By category:
  • Maintenance updates: https://pfcomputing.com/categories/maintenance/index.xml
  • Feature updates: https://pfcomputing.com/categories/feature/index.xml

For subscription options via Slack channels, please refer to Add RSS feeds to Slack | Slack.

Email Newsletter

Currently under preparation.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

PFCP Tutorial: Deploying Workloads

This page guides you through deploying server workloads using a cluster and introduces you to monitoring services.

Preliminary Setup

Complete the following steps in the left column under “Connect to Cluster”:

1. You are logged in to the PFCP portal.
2. Your cluster connection has been configured.

Creating a Namespace

Create a Namespace¹ that will be used throughout this tutorial.

Note

Creating Namespaces requires “Organization Administrator” privileges. Regular users will encounter failure when attempting to create a Namespace.

For users with standard privileges, please request assistance from your organization’s Organization Administrator to create a Namespace.

1. Navigate to the Namespaces page in the portal.

2. Open the Create Namespace interface.

   • Cluster Name: Select the cluster you intend to use.
   • Namespace Name: org-<organization-name>--<any-unique-value> ²

3. Click the Create button to initiate the Namespace creation process.

4. Set the created Namespace as the default Namespace for kubectl commands.

   $ kubectl config set-context --current --namespace=<your-created-namespace-name>
   

5. Verify that the Namespace has been created and that the configuration is correct. Run kubectl get pod - if no errors occur, the configuration is successful. If errors appear, first confirm the Namespace exists, then double-check that your execution command is correct.

   // Successful configuration case
   $ kubectl get pod
   No resources found in <your-created-namespace-name> namespace.
   
   // Configuration error case
   $ kubectl get pod
   Error from server (Forbidden): pods is forbidden: User "oidc:org-<organization-name>/<username>" cannot list resource "pods" in API group "" in the namespace "<your-created-namespace-name>"
   

Deploying Workloads

We will execute a sample Pod using PFCP’s Kubernetes cluster³. Additionally, we will expose the deployed Pod to the internet and verify that it can be accessed via a web browser.

For this example, we are using the podinfo container image.

1. Execute the following command to create a Deployment that runs podinfo:

   $ kubectl create deployment podinfo --image=stefanprodan/podinfo --port=9898
   deployment.apps/podinfo created
   

2. Verify that the Pod has started successfully.

   $ kubectl get pod
   NAME                       READY   STATUS    RESTARTS   AGE
   podinfo-554c877494-p58gf   1/1     Running   0          25s
   

3. Next, create a Service to expose the running Pod.

   $ cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: podinfo
     name: podinfo
   spec:
     ports:
     - name: http
       port: 8080
       protocol: TCP
       targetPort: 9898
     selector:
       app: podinfo
   EOF
   service/podinfo created
   
   $ kubectl get service
   NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   podinfo   ClusterIP   10.100.212.250   <none>        8080/TCP   77s
   

4. To make the created Service publicly accessible via the internet, create an Ingress.

   $ HOST=podinfo-<any-desired-name>.<organization-name>.sr1-01.ingress.pfcomputing.com
   $ kubectl create ingress podinfo-ingress --class=nginx --rule="${HOST}/*=podinfo:8080"
   ingress.networking.k8s.io/podinfo-ingress created
   
   $ kubectl get ingress
   NAME              CLASS   HOSTS          ...
   podinfo-ingress   nginx   <same-value-as-HOST>   ...
   

5. Access https://<same-value-as-HOST> in your web browser. If the podinfo dashboard appears, the deployment is successful.

Monitoring Verification

PFCP provides Grafana and Prometheus as managed services. We will use these to check the resource usage status of our created Pod.

1. Access the Grafana dashboard. You can reach it via the link displayed on the portal’s home page.

2. From the hamburger icon in the top-left corner, navigate to Dashboards > kube-prometheus > Kubernetes / Compute Resources / Pod to view Pod resource usage. Select the appropriate Namespace and Pod name from the dropdown menu. You should see a screen similar to the following:

   

3. You can also collect and visualize metrics instrumented in the Pod using Prometheus. As an example, let’s collect metrics for the podinfo Pod created above⁴.

   Create a ServiceMonitor custom resource as follows:

   $ cat << EOF | kubectl apply -f -
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: podinfo
   spec:
     selector:
       matchLabels:
         app: podinfo
     endpoints:
     - interval: 30s
       port: http
       path: /metrics
   EOF
   servicemonitor.monitoring.coreos.com/podinfo created
   

4. Verify that the podinfo scraping was successful and metrics are being collected. Access Prometheus’s web UI and select Status > Targets from the top tab. If you see <your-created-namespace-name>/podinfo/0 listed in the targets with a status of Up, the scraping is functioning correctly.

   

5. The collected metrics can be visualized in Grafana. Open the Grafana dashboard and select Explore from the hamburger icon in the top-left corner. For example, running the following PromQL query will display a time-series graph showing the number of times the /metrics endpoint was called.

   sum by (pod) (rate(promhttp_metric_handler_requests_total{namespace="<your-created-namespace>", job="podinfo"}[$__rate_interval]))
   

   

1. In PFCP, the created Namespace will be treated as a sub-namespace. For detailed information, please refer to the Glossary. ↩

2. The suffix must be different from any existing Namespaces. When conducting this tutorial with multiple users, we recommend using usernames as suffixes. ↩

3. For detailed information on workload deployment, please refer to the official documentation. ↩

4. podinfo provides Prometheus-compatible metrics via /metrics. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

PFCP Tutorial: Using MN-Core with PyTorch

This document explains how to use MN-Core through the Machine Learning Software Development Kit (MLSDK).

What is MLSDK?

MLSDK is a software development environment that includes compilers, runtime software stacks, and documentation to enable the use of MN-Core with PyTorch. While its name includes “Machine Learning” (ML), it can be equally used for developing high-performance computing software in domains beyond machine learning.

Setting Up Your Environment

To configure your environment, use the “Workspace” feature to create an interactive development environment on your cluster.

1. Navigate to the Workspace page in the portal and click the Create New button.
2. Fill out the form and click the Create button.

Note

Add Persistent Storage

Any file modifications made to paths without mounted persistent storage will be lost and not persisted. To ensure your changes are preserved, allocate new storage from “Add Persistent Storage,” mount it to a path like /data, and save your files there.

Accessing Your Environment

Access your development environment (JupyterLab) by clicking the link in the URL column of your created workspace.

Note

Creating workspaces may take some time.

If creation doesn’t complete after a reasonable period, check that the values entered in the form were correct. Try creating it again.

Open a terminal by clicking the “Launcher → Other / Terminal” button in JupyterLab.

In the terminal, run the following command to verify the MN-Core 2 devices connected to your environment (the output will vary depending on the allocated devices):

$ gpfn3-smi list
0: mnc2p28s0

If you don’t see any output, double-check your workspace configuration for any errors.

Starting the MLSDK Tutorial

Refer to the MLSDK documentation to begin the tutorial. The MLSDK tutorial documentation is also included in the container image, so you can use that version as well.

$ cat /opt/pfn/pfcomp/codegen/MLSDK/README.md

Deleting Your Environment

After you no longer need your environment, delete it from the portal page.

1. Access the Workspace page in the portal.
2. Click Delete from the ⋯ button for the workspace you wish to remove.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

PFCP Tutorial: Importing Data from Cloud Storage to Your Cluster

This page provides step-by-step instructions for transferring data from cloud storage to PFCP.

As an example, we will demonstrate how to copy data from an AWS S3 bucket managed by the user to a Persistent Volume within PFCP.

Preliminary Setup

Please complete the following steps from the “Connect to Your Cluster” section in the left column:

1. You are logged in to the PFCP portal
2. Your cluster connection has been properly configured

Additionally, you will need an AWS account and the target S3 bucket where you want to synchronize your data¹.

Configuring Public Cloud Identity Federation and Copying Data

AWS Configuration

These instructions assume you are using the sr1-01 region. If you are in a different region, please adjust the domain names accordingly.

1. Create an OIDC provider within the AWS account you wish to access².

   • For the provider (issuer) URL, refer to the Regions and Compute Nodes documentation.
   • Allowed audience: sts.amazonaws.com

2. Create an IAM role that will be used to access AWS. This IAM role will be linked to a Kubernetes ServiceAccount through identity federation. Here, we will create an IAM role named data-transfer-sr1-01.

3. Configure a trust policy that specifies the Kubernetes ServiceAccount to which this role should be bound for identity federation³. You can either use an existing ServiceAccount or create one later. For this example, we will create a ServiceAccount named data-transfer-sa.

   {
       "Version": "2012-10-17",
       "Statement": [{
           "Sid": "",
           "Effect": "Allow",
           "Principal": {
               // Specify the ARN of the oidc provider created in the previous step.
               "Federated": "arn:aws:iam::{aws_account_id}:oidc-provider/token.sr1-01.kubernetes.pfcomputing.com"
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
               "StringEquals": {
                   // Specify the name of the Kubernetes ServiceAccount to which this role should be bound.
                   "token.sr1-01.kubernetes.pfcomputing.com:sub": "system:serviceaccount:<namespace>:data-transfer-sa"
               }
           }
       }]
   }
   

4. Configure a policy on the S3 bucket containing the data you wish to transfer to allow access from a specific IAM role⁴.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::{aws_account_id}:role/data-transfer-sr1-01"
         },
         "Action": [
           "s3:ListBucket",
           "s3:GetObject",
         ],
         "Resource": [
           "arn:aws:s3:::example-bucket",
           "arn:aws:s3:::example-bucket/*"
         ]
       }
     ]
   }
   

Kubernetes Cluster Configuration

1. Create a ServiceAccount. Specify the AWS IAM role data-transfer-sr1-01 with access to your S3 bucket in the annotations of the ServiceAccount you will use for data copying.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: data-transfer-sa
     annotations:
       aws.id-federation.preferred.jp/role-arn: "arn:aws:iam::{aws_account_id}:role/data-transfer-sr1-01"
   

2. Verify that identity federation is functioning correctly. When creating a Pod, specify this ServiceAccount in the spec.serviceAccountName field. The Pod will automatically be configured with an AWS session token valid for one hour. When the token expires, the AWS SDK will automatically refresh it. Run the aws sts get-caller-identity command to verify that the session token can successfully assume the AWS IAM Role bound to the ServiceAccount.

   $ kubectl run --rm --overrides='{"spec":{"serviceAccountName": "data-transfer-sa"}}' id-federation-check --image=amazon/aws-cli -- sts get-caller-identity
     {
         "UserId": "***********************:botocore-session-1751604047",
         "Account": "{aws_account_id}",
         "Arn": "arn:aws:sts::{aws_account_id}:assumed-role/data-transfer-sr1-01/botocore-session-1751604047"
     }
   

3. Create a Persistent Volume Claim (PVC) to store your data⁵.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: data-transfer-pvc
   spec:
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard-rwx-<organization-name>
   

4. Create a Job to copy data using the AWS CLI’s s3 sync command. Mount the PVC created above and specify data-transfer-sa in the template.spec.serviceAccountName field, ensuring it has enabled identity federation with AWS. Replace <bucket-name> and <object-name> with appropriate values.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: data-transfer-job
   spec:
     template:
       spec:
         serviceAccountName: data-transfer-sa # Specifies the ServiceAccount configured for identity federation
         containers:
         - name: transfer
           image: amazon/aws-cli
           command: ["aws"]
           args: ["s3", "sync", "s3://<bucket-name>/<object-name>", "/mnt/data"]
           volumeMounts:
           - mountPath: "/mnt/data"
             name: my-volume
         volumes:
         - name: my-volume
           persistentVolumeClaim:
             claimName: data-transfer-pvc
         restartPolicy: OnFailure
   

   Run kubectl logs job/data-transger-job to verify that the command completed successfully.

   $ kubectl logs job/data-transger-job
       download: s3://<bucket-name>/file1.txt to /mnt/data/file1.txt
       ...
       Completed 10 of 10 file(s), 100% done.
   

Cleanup

Delete all resources created during this tutorial.

kubectl delete job/data-transfer-job
kubectl delete pvc/data-transfer-pvc
kubectl delete serviceaccount/data-transfer-sa

If necessary, also delete the corresponding AWS resources.

1. Getting Started with Amazon S3 ↩

2. Creating an OpenID Connect Identity Provider in IAM - AWS Identity and Access Management ↩

3. Creating a Role for OpenID Connect Federation ↩

4. Amazon S3 Policies and Permissions ↩

5. Using File Storage ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

MN-Core SDK

The MN-Core SDK is a development framework for creating software that utilizes the MN-Core series. It consists of two SDKs: MLSDK and HPCSDK.

• MLSDK: An SDK for integrating MN-Core series functionality into PyTorch programs.
• HPCSDK: An SDK for incorporating MN-Core series capabilities into C/C++ programs.

For documentation on each of these SDKs, please refer to the following resources:

MLSDK

• MLSDK Documentation (Japanese)
• MLSDK Documentation (English)

HPCSDK

Coming soon.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Roles in PFCP

This section describes the available roles within PFCP and the operations permitted for each role.

Role Types

PFCP supports two primary role types: Organization Administrator and Regular User.

Organization Administrator

• Has full administrative privileges over all operations within their organization.
• Possesses equivalent admin permissions to the Kubernetes standard roles for the root namespace and all sub-namespaces of the Kubernetes cluster.

Regular User

• Can utilize the Kubernetes cluster associated with their organization.
• Has equivalent edit permissions to the Kubernetes standard roles for the root namespace of the Kubernetes cluster.
• By default, has no permissions for sub-namespaces. Additional permissions must be granted through RoleBindings.
• Cannot perform user management operations.

Operation Summary

Cluster Usage

User Management

1. Unlike standard Kubernetes admin roles, certain resource permissions have been modified to remove some privileges while adding permissions for custom resources used by PFCP. This role is defined through the org-admin ClusterRole. For detailed permission specifications, please refer to the documentation pages for each resource. ↩ ↩2

2. Similar to org-admin, these permissions differ partially from standard Kubernetes edit roles. They are defined through the org-edit ClusterRole. For detailed permission specifications, please refer to the documentation pages for each resource. ↩

3. The granted permissions vary depending on the Role associated with RoleBindings. For detailed information on RoleBindings, please refer to Permission Configuration (RBAC). ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Managing Organizational Users

In PFCP, organization administrators are responsible for managing users within their organizational units.

PFCP supports two primary methods for user management:

1. Direct user management through the portal
2. Group-based management through integration with external authentication providers

User Management

This section explains how to directly invite users, modify their roles, or delete them from your organization.

Note

Only Google personal accounts or Google Workspace accounts can be used for direct invitations.

Warning

Changes to user roles or user groups may take some time to propagate to currently logged-in user sessions. Please try logging out and back in, or clearing your browser cookies to resolve this issue.

Inviting Users

1. Navigate to the User Management page in the portal and click the Send Invitation Email button.
2. Enter the Email address and role for the user you wish to invite, then click the Send Invitation Email button.
3. The sent invitation will appear on the Pending Acceptance List. It will disappear from the list once the invitation is accepted.
4. If the invitation expires, please resend it.

Changing User Roles

1. Access the User Management page in the portal and view the User List.
2. Select the user whose role you wish to modify and change their role.
   • Note: You cannot change your own role while logged in.

Deleting a User from the Organization

1. Access the User Management page in the portal and view the User List.
2. Select the user you wish to remove and delete them from the organization.
   • To restore a deleted user, you must re-invite them.
   • You cannot delete the currently logged-in user.

Managing User Groups

You can organize organizational users into groups for more efficient namespace permission management. By integrating user groups with external authentication providers, you can also link accounts from your external authentication system to your organization.

For permission management details, please refer to Permission Settings (RBAC).

Joining User Groups

Users can join user groups in two ways - both methods will effectively add them to the group.

• Direct group membership
  • Add users to user groups through the portal interface.
• Integration with external authentication groups
  • By linking user groups with groups from external authentication providers, users belonging to those external groups will be automatically added to the user group.

Creating a User Group

1. Access the User Group Management page in the portal and click the Create New button.
2. Fill out the form and click the Create button.

Deleting a User Group

1. Access the User Group Management page in the portal.
2. Select the group you wish to delete and click the Delete button.

Adding Users to a User Group

1. Access the User Group Management page in the portal and select the group you wish to edit.
2. In the Add Members section, select the user(s) you wish to add and click the Add button.
   • Note: The user(s) must first be invited and accepted into your organization.

Note

It may take up to one hour for group membership changes to reflect in your Kubernetes cluster connection information.

Removing Users from a User Group

1. Access the User Group Management page in the portal and select the group you wish to edit.
2. In the Members section, select the user(s) you wish to remove and click the Remove Selected Members button.

Creating Integrations Between User Groups and External Authentication Providers

1. Access the External Authentication Integration page in the portal.
2. Enter the External Authentication Group ID and PFCP Group Name, then click the Add Group Integration button.
   • Note: The PFCP group must be created in advance.

Deleting a User Group Integration

1. Access the External Authentication Integration page in the portal.
2. Select the integration you wish to delete and click the Remove Selected Group Integration button.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Logging In to the PFCP Portal

PFCP provides a portal for obtaining authentication credentials to connect to Kubernetes clusters and for managing various administrative functions.

First-time Login

Note

This section describes the standard login method using your email address. Note: The login process may vary depending on your organization. For details, please consult with your organization administrator.

1. Request a user invitation from your organization administrator.
2. Once you receive the invitation email, click the ACCEPT INVITATION button in the email body.
3. A confirmation screen for the invitation will appear. Verify that the organization name and inviter are correct, then click Continue.
4. Log in using the email address you received the invitation email from.
5. After logging in, you will be prompted to enter the organization name; enter the name you were invited to.
6. A final verification of your login status will occur, after which you will be automatically redirected to the portal.

Subsequent Logins

1. Access the Portal.
2. A screen will appear for entering the organization name; enter the name associated with your account.
3. Log in using your registered email address.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Connecting to a Cluster

This page provides instructions for connecting to your Kubernetes cluster.

Installing the Command Line Tool for Cluster Operations

You will connect to your Kubernetes cluster using the kubectl command-line tool.

For installation instructions, refer to the official documentation.

Configuring Kubernetes Cluster Connection Credentials

1. Navigate to the kubectl setup page in your portal.
2. From the Cluster Name dropdown menu, select the cluster you wish to use.
3. Click Get Credentials, copy the generated command, and execute it in your terminal.

Note

Connecting to Clusters from Multiple Environments

If you are connecting to clusters from multiple environments, repeat the Get Credentials process for each environment to generate and use separate commands.

Reusing the same credentials across multiple environments will result in credential invalidation and failure to operate the cluster, accompanied by the following error:

failed to refresh token: oauth2: "invalid_grant" "Unknown or invalid refresh token."

If your credentials have been invalidated, you will need to obtain new credentials and re-configure them.

Verifying the Connection

Run kubectl auth whoami to verify that your Username displays your email address and that the Groups section shows your organization’s group name.

$ kubectl auth whoami
ATTRIBUTE   VALUE
Username    oidc:<your-organization-name>/<your-email-address>
Groups      [oidc:<your-organization-name> system:authenticated]

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Learning the Fundamentals of Kubernetes

PFCP is built on Kubernetes, an open-source container orchestration platform. The documentation on this site assumes familiarity with basic Kubernetes concepts and terminology. If you’re new to Kubernetes, please review these fundamentals here. Additionally, gaining deeper knowledge of Kubernetes through official documentation or books will help you make more effective use of PFCP.

Overview of Kubernetes

Kubernetes is an open-source container orchestration platform hosted by the Cloud Native Computing Foundation (CNCF). It provides automated tools for deploying, scaling, and managing the availability of containerized applications, and has gained widespread adoption as the standard in this field.

For detailed information, please refer to the official Kubernetes documentation.

• Overview | Kubernetes

Note

PFCP is configured as a cloud service for deep learning and AI workloads based on Kubernetes. The clusters we provide are managed, meaning users do not need to operate the clusters themselves.

Clusters and Nodes

A cluster refers to a collection of machines that execute and manage workloads. From a user’s perspective, it can be treated as a single unified computing resource.

A node represents an individual machine that constitutes a cluster. In practice, this typically refers to either a standard virtual machine or a physical machine.

When a user instructs Kubernetes to deploy a workload, the system examines the resource requirements specified in the workload definition—including CPU, memory, and device specifications like MN-Cores—and then schedules and executes the workload on the appropriate node.

• Kubernetes Components | Kubernetes
• Cluster Architecture | Kubernetes
• Nodes | Kubernetes

Note

In PFCP terminology, a node that runs user workloads is referred to as a “compute node.”

Namespace

A Kubernetes Namespace is a mechanism for logically partitioning a single cluster. Within Kubernetes, resource names must be unique within their respective Namespaces. Namespaces facilitate resource management by allowing separation of environments like staging and production, or by enabling shared usage among multiple teams or projects.

For detailed information, please refer to the official Kubernetes documentation.

• Namespaces | Kubernetes

Note

Typically, Kubernetes Namespace management requires cluster-wide administrator privileges. However, in PFCP, organizational administrators can manage Namespaces using Kubernetes custom add-ons.

Pod

In Kubernetes, workloads are organized into Pods, which consist of one or more containers. A Pod represents the smallest deployable unit in Kubernetes. Containers within a single Pod must always execute on the same node, sharing both network and storage resources while operating together.

For detailed information, please refer to the official Kubernetes documentation.

• Pods | Kubernetes

Containers and Container Images

Containers are a technology that enables the execution of isolated processes separate from the rest of the system. Since containers operate independently from the host and other processes, they are less prone to environment-dependent issues and can be executed consistently across different environments.

The necessary code, libraries, and runtime required to run containers must be packaged into a container image. These container images are stored and distributed through a service called a container image registry.

For detailed information, please refer to the official Kubernetes documentation.

• Containers | Kubernetes

Note

The PFCP dedicated container image registry provides container images containing all required files and software for MN-Core deployment, allowing you to quickly begin using MN-Core.

Note

PFCP does not provide its own container image registry for users to build and use container images in their clusters. Please use an external container image registry service instead.

Higher-level resources managing Pods: Deployment, StatefulSet, Job, CronJob…

Kubernetes offers higher-level resource types to manage Pods, supporting various types of workloads.

• Deployment: Runs multiple Pod replicas distributed across cluster nodes. It automatically replaces failed or non-responsive Pods, making it ideal for maintaining highly available services.
• StatefulSet: Runs Pod replicas with unique identifiers for each instance. Particularly useful for running stateful applications.

• Job: Creates and retries running Pod replicas until a specified number of Pods successfully complete execution. Suitable for one-time training tasks.

• CronJob: Creates Kubernetes Jobs based on specified schedules, similar to Linux cron.

  Warning

  Pods created directly by users without using higher-level resources (Bare Pods) will not be automatically rescheduled on other nodes if they stop due to node failures. Generally speaking, avoid creating Pods directly and instead use higher-level resources.

For detailed information, please refer to the official Kubernetes documentation.

• Deployments | Kubernetes

• StatefulSets | Kubernetes

• Jobs | Kubernetes

• CronJob | Kubernetes

  Note

  PFCP also provides custom resources that are not part of Kubernetes but are particularly useful for running deep learning and AI workloads. For information about custom resources, please refer to this user guide.

Service

A Service provides a fixed network endpoint for a set of Pods. Since Pods automatically receive different IP addresses each time they are launched, you cannot reliably access them directly.

Additionally, Kubernetes includes built-in DNS functionality for address resolution within the cluster and provides service discovery capabilities using Service names. This allows you to access your workloads reliably by referencing the Service name.

For detailed information, please refer to the official Kubernetes documentation.

• Service | Kubernetes

Ingress

An Ingress provides HTTP/HTTPS access to Kubernetes Services from outside the cluster. It enables request routing based on hostnames and paths.

For detailed information, please refer to the official Kubernetes documentation.

• Ingress | Kubernetes

Note

In PFCP, you can use Ingress to securely expose services to the internet. See Publishing Workloads as Web APIs or Publishing Workloads as Web Applications for details.

Persistent Storage

Data stored in running Pods is lost when the Pod is terminated. For data that needs to be retained beyond the lifecycle of a Pod, you can provision persistent storage using PersistentVolumeClaims and mount it to your Pods for use.

For detailed information, please refer to the official Kubernetes documentation.

• Persistent Volumes | Kubernetes

Manifest Files

YAML or JSON files that define the configuration state of resources you want to create and manage in your cluster are called manifest files. They are typically written in YAML format.

Kubernetes provides two methods for deploying workloads to your cluster: imperative creation/update via command-line tools, and declarative creation/update using manifest files. The imperative method offers the advantage of quick operations for temporary use. However, due to its lower reproducibility, we recommend using the declarative method with manifest files, which allows you to manage desired states as files—a practice that is generally preferable in most scenarios. For detailed information, please refer to the official Kubernetes documentation.

• Managing Workloads | Kubernetes

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Logical Cluster Division Using Namespaces

Kubernetes enables the creation of multiple workspaces by defining Namespace resources, allowing independent management of resources within each workspace. This feature proves particularly useful when operating environments with different access permissions or network connectivity requirements, such as development and production environments, or when multiple teams need to share resources.

PFCP provides functionality for creating Namespace resources. However, because PFCP’s Kubernetes clusters are configured as multi-tenant environments, there are the following constraints on Namespace resource creation that differ from standard Kubernetes clusters:

Warning

Namespace Constraints

• Each organization is automatically provisioned with one Namespace resource as its root namespace.
• Additional Namespace resources must be created as subordinate subnamespaces¹ under this root namespace.
• The resource name for the root namespace is org-<organization-name>.
• For subnamespaces, the resource name must start with the prefix org-<organization-name>--.
• The root namespace is pre-created across all PFCP clusters and cannot be deleted.
• Subnamespaces must be created individually for each PFCP cluster and can be deleted.
• Direct creation of Namespace resources using tools like kubectl is not allowed.²
• You cannot create subnamespaces that themselves contain further subnamespaces (grandchild namespaces).

Below we explain how to manage subnamespaces.

Creating Subnamespaces

Follow these steps to create a subnamespace:

1. Access the Namespaces page in the portal.
2. Open the Create Namespace interface.
3. Enter the Cluster Name, Namespace Name, and an optional description.
4. If you do not want communication from other Namespaces within your organization to be permitted, check the optional² box.
5. Click the Create button to initiate the subnamespace creation process.

Once created, the subnamespace can be used as a standard Namespace resource.

# Example: After creating a subnamespace `org-<organization-name>--foo`
$ kubectl get all -n org-<organization-name>--foo
No resources found in org-<organization-name>--foo namespace.

Organization administrators are automatically granted org-admin role permissions for the subnamespaces they create. Regular users do not receive automatic permissions for subnamespaces; you must create a RoleBinding to assign them permissions. For details on RoleBindings, see Permission Configuration.

Modifying Subnamespaces

1. Access the Namespaces page in the portal.
2. Open the modification interface for the subnamespace you wish to change and make your adjustments. Note that you cannot modify the cluster name or namespace name.
3. Click the Update button to apply the changes to the subnamespace.

Note

The root namespace cannot be modified.

Deleting Subnamespaces

1. Access the Namespaces page in the portal.
2. Select the Namespace you want to delete and remove it.

Warning

When deleting a subnamespace, all resources created within that Namespace resource will also be removed.

Note

The root namespace cannot be deleted.

Reference: Managing Subnamespaces via Command Line Tools

The subnamespace functionality is implemented through the SubnamespaceAnchor custom resource of Hierarchical Namespace. By installing the kubectl-hns plugin, you can perform subnamespace operations directly from the terminal.

Creating Subnamespaces

# Create a subnamespace `org-<organization-name>--foo`
$ kubectl hns create org-<organization-name>--foo -n org-<organization-name>
Successfully created "org-<organization-name>--foo" subnamespace anchor in "org-<organization-name>" namespace

Viewing Hierarchical Structure

$ kubectl hns tree org-<organization-name>
org-<organization-name>
└── [s] org-<organization-name>--foo

Deleting Namespaces

$ kubectl delete subnamespaceanchor org-<organization-name>--foo -n org-<organization-name>

1. To implement subnamespace functionality, we are utilizing Hierarchical Namespace. ↩

2. By using the SubnamespaceAnchor custom resource provided by Hierarchical Namespace, you can create Namespace resources as subnamespaces. ↩ ↩2

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Managing User Cluster Access Permissions

PFCP supports Role-Based Access Control (RBAC) using Kubernetes’ Role/RoleBinding mechanism. By creating a RoleBinding, you can assign permissions defined in Roles to specific users or groups.

Note

Before proceeding, please review the PFCP Roles documentation to understand the roles available in PFCP.

Standard Roles and Groups Provided by PFCP

The following three ClusterRoles¹ are available:

• org-view
  • Provides the necessary permissions to view user workloads.
• org-edit
  • In addition to the org-view permissions, includes the necessary permissions to execute user workloads.
• org-admin
  • In addition to the org-edit permissions, includes administrative privileges such as Role/RoleBinding management.

Tip

Each ClusterRole is based on the standard Kubernetes view/edit/admin ClusterRoles, with additional permissions removed for some resources and added permissions for custom resources used in PFCP. You can check exactly which operations are permitted on which resources using the following command:

$ kubectl get clusterrole org-view org-edit org-admin -o yaml

Additionally, the following two groups are pre-configured:

• org-<organization-name> (e.g., org-pfn)
  • Both organization administrators and regular users belong to this group.
  • Only the org-edit Role is granted for the root Namespace. Roles are not granted for any subNamespaces.
• org-<organization-name>/admin (e.g., org-pfn/admin)
  • This group is for organization administrators.
  • The org-admin ClusterRole is granted for both the root Namespace and all subNamespaces.

The root Namespace can be used as is with default settings because all users belonging to the organization have org-edit or higher permissions². For newly created subNamespaces, regular users do not have any permissions. To grant permissions to regular users, you must create a RoleBinding.

Creating RoleBindings

RoleBindings can be created in one of two ways:

1. Creating a RoleBinding for a group
2. Creating a RoleBinding for a specific user

Below we explain how to grant org-edit ClusterRole² to both groups and users for subNamespaces.

Creating a RoleBinding for a Group

To grant permissions to all users for a subNamespace (org-<organization-name>--foo), create the following RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: org-edit
  namespace: org-<organization-name>--foo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: org-edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: oidc:org-<organization-name>

You can also grant permissions to user groups created through Managing Organization Users. To grant permissions to a user group (ops) for a subNamespace (org-<organization-name>--foo), create the following RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: org-edit-ops # The name can be arbitrary
  namespace: org-<organization-name>--foo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: org-edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: oidc:org-<organization-name>/ops  # Specify the PFCP user group name

Note

If you would like to enable SAML integration with an external identity provider, please contact support for assistance.

Creating a RoleBinding for a Specific User

To grant permissions to a specific user (alice@example.com) for a subNamespace (org-<organization-name>--foo), create the following RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: org-edit-user-alice  # The name can be arbitrary
  namespace: org-<organization-name>--foo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: org-edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: oidc:org-<organization-name>/alice@example.com

Removing Workload Execution Permissions for Regular Users on Root Namespaces

As noted in the Standard Roles and Groups Provided by PFCP, regular users are by default granted org-edit permissions on the root Namespace. To remove org-edit permissions for regular users on the root Namespace, execute the following command:

$ kubectl -n org-<organization-name> delete rolebindings org-edit

This will prevent regular users from creating any resources or executing workloads in the root Namespace.

Note

Regular users will always retain org-view permissions on the root Namespaceand this cannot be disabled by organization administrators.

Creating Custom Roles

The org-admin ClusterRole includes permissions to create custom Roles within Namespaces, allowing organization administrators to create and assign Roles with custom permission sets to regular users.

For example, to create a custom Role that only grants view permissions for Pods, use the following configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-view
  namespace: org-<organization-name>--foo
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

Related Links

• Using RBAC Authorization | Kubernetes

Frequently Asked Questions

Q. As an organization administrator, I want to use the org-edit role when interacting with clusters

The org-admin role granted to organization administrators includes powerful permissions for managing subNamespaces and RoleBindings. For executing training workloads, using the org-edit role helps prevent accidental modifications. This can be achieved by creating a ServiceAccount with the org-edit role and operating as that ServiceAccount.

Below are the steps to operate the org-<organization-name>--foo subNamespace using the org-edit role:

// Create a ServiceAccount in the `org-<organization-name>--foo` Namespace.
$ kubectl -n org-<organization-name>--foo create sa org-edit-sa
serviceaccount/org-edit-sa created

// Grant the `org-edit` role to the created ServiceAccount.
$ kubectl -n org-<organization-name>--foo create rolebinding org-edit-sa --clusterrole=org-edit --serviceaccount=org-<organization-name>--foo:org-edit-sa
rolebinding.rbac.authorization.k8s.io/org-edit-sa created

// Use the --as system:serviceaccount:<Namespace>:<ServiceAccount> flag to impersonate the target ServiceAccount.
// Verify that you are impersonating the correct ServiceAccount.
$ kubectl --as system:serviceaccount:org-<organization-name>--foo:org-edit-sa auth whoami
ATTRIBUTE   VALUE
Username    system:serviceaccount:org-<organization-name>--foo:org-edit-sa
Groups      [system:serviceaccounts system:serviceaccounts:org-<organization-name>--foo system:authenticated]

// While the `org-admin` role would allow creating ResourceQuotas, impersonation prevents this operation.
$ kubectl auth can-i create resourcequotas -n org-<organization-name>--foo
yes
$ kubectl --as system:serviceaccount:org-<organization-name>--foo:org-edit-sa auth can-i create resourcequotas -n org-<organization-name>--foo
no

If you prefer not to repeatedly add the --as flag, you can configure the following in the user field of your kubeconfig file to set it as the default:

 - name: pfcp-<organization-name>-<cluster-name>
   user:
+    as: system:serviceaccount:<Namespace>:<ServiceAccount>
     auth-provider:
       config

1. A ClusterRole is a role that can be used from any Namespace. ↩

2. If you do not need to grant the org-edit ClusterRole for root Namespaces to regular users, you can opt out. For instructions, see Removing Workload Execution Permissions for Regular Users on Root Namespaces. ↩ ↩2

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Configuring Inter-Namespace Communication Controls

In PFCP, inter-namespace communication is managed according to the following policies:

Warning

Inter-Namespace Communication Policy

• All communication across Namespaces belonging to different organizations is strictly prohibited.
• By default, all communication between Namespaces within the same organization is allowed.
• Custom rejection policies can be added as needed. Communication that is already blocked by PFCP services cannot be permitted.

To implement custom rejection policies, use the CiliumNetworkPolicy custom resource^{1 2}. Below is an example demonstrating how to configure a policy to block ingress traffic from external Namespaces.

Blocking Ingress Traffic from External Namespaces Using CiliumNetworkPolicy

By creating a CiliumNetworkPolicy resource within the Namespace you wish to isolate, you can block traffic from other Namespaces. This proves particularly useful when running production workloads in a Namespace and wanting to restrict access from other Namespaces.

For example, by creating the following resource in the org-<organization-name> Namespace, only traffic originating from within the same Namespace or from the org-<organization-name>--foo Namespace will be permitted:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: "deny-except-org-<organization-name>--foo"
  namespace: "org-<organization-name>"
spec:
  endpointSelector: {}
  ingressDeny:
  - fromEndpoints:
    - matchExpressions:
      - key: k8s:io.kubernetes.pod.namespace
        operator: NotIn
        values: ["org-<organization-name>", "org-<organization-name>--foo"]

In addition to spec.ingressDeny, you can also use spec.egressDeny to block egress traffic to specific destinations. For cases where you want to prohibit outgoing traffic to particular destinations, please utilize spec.egressDeny.

Note that spec.ingress and spec.egress cannot be used. Communication with blocked destinations cannot be permitted.

Options for Blocking Ingress Traffic When Creating Sub-Namespaces

When creating or modifying sub-Namespaces through the portal interface, you can enable an option to create a CiliumNetworkPolicy that blocks all incoming traffic from other Namespaces. This feature is particularly useful for environments running production workloads where access from other Namespaces should be restricted.

This CiliumNetworkPolicy will be created with the name deny-all-ingress. After creation, you can freely modify and adjust the policy as needed.

1. Certain features of CiliumNetworkPolicy are restricted. ↩

2. For detailed documentation on CiliumNetworkPolicy, please refer to the official documentation. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Monitoring Organizational Resource Usage

PFCP implements quotas at the organizational level rather than per Namespace, applying a limit to the total resource usage across all Namespaces within an organization. Attempting to create resources beyond this limit will result in creation failures.

You can check both the total used resources and the usage limits from your organization’s Resource Quota page in the portal.

Note

If you need to request quota relaxation, please contact our support team.

Checking Resource Quotas

To manage resource quotas across organizational Namespaces collectively, PFCP utilizes the Hierarchical Resource Quota feature. You can view the hierarchical resource quota limits and usage status through the following methods:

Checking Resource Quotas via the Grafana Dashboard

PFCP provides a Grafana dashboard for monitoring hierarchical resource quotas. In the Grafana dashboard here, navigate to “PFCP > HNC / HRQ / Tenant Resource Hard Limit & Usage” to view your organization-wide resource quotas.

Checking Resource Quotas Using Command Line Tools

By installing the kubectl-hns plugin, you can check resource quotas from your terminal.

The following command allows you to view your organization’s total resource usage status:

$ kubectl get hrq -o yaml -n org-<organization-name>

The output will include two key fields: .status.hard and .status.used.

• .status.hard: Indicates the maximum number of resources that can be created under the specified Namespace. You cannot create more resources than this limit.
• .status.used: Shows the total count of resources that have already been created under the specified Namespace.

$ kubectl get hrq org-resource-quota -o yaml -n org-<org-name>
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchicalResourceQuota
metadata:
  name: org-resource-quota
  namespace: org-<org-name>
spec:
  hard:
    count/configmaps: "100"
    count/cronjobs.batch: "100"
    ...
status:
  hard:
    count/configmaps: "100"
    count/cronjobs.batch: "100"
    ...
  used:
    count/configmaps: "3"
    count/cronjobs.batch: "0"
    ...

Related Links

• Resource Quotas | Kubernetes

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Limiting Resource Usage by Namespace

You can restrict resource usage on a per-Namespace basis by creating a ResourceQuota resource. ^{1 2} This allows you to prevent excessive consumption of compute resources within specific Namespaces and to balance resource allocation across different Namespaces. ³

To set quotas for the subNamespace org-<organization-name>--foo, you would create a ResourceQuota with the following configuration:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota
  namespace: org-<organization-name>--foo
spec:
  hard:
    requests.preferred.jp/mncore2: 4
    requests.nvidia.com/gpu: 0

In this example, the quota limits MN-Core 2 usage to a maximum of 4 units and restricts NVIDIA GPU usage.

Related Links

• Resource Quotas | Kubernetes

1. ResourceQuotas by Namespace are not automatically created. You must create them as needed. ↩

2. ResourceQuotas can also be created for the root Namespace. ↩

3. If you set a quota value larger than the organization-wide quota, the organization-wide quota will take precedence. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Expanding Storage Quotas

All organizations are initially allocated up to 256GiB of storage and best-effort I/O bandwidth of 32MB/s, 8K IOPS.

If you require additional storage quota, please contact our support team.

Frequently Asked Questions

Q. My available quota should be sufficient, but I’m still encountering failures when provisioning persistent storage

Removing a PersistentVolumeClaim will also automatically delete any dynamically provisioned PersistentVolume. The reclaimed space will become available again after 12 hours. Please note that while the quota display updates immediately, the space remains unavailable for reuse until 12 hours have passed.

Note

If you need to urgently restore storage quota after deleting a PersistentVolume, please contact our support team immediately.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Configuring GitOps-Style Continuous Delivery

GitOps is a methodology that enables declarative management of resource configurations using Git, with changes automatically applied to the cluster. By adopting GitOps, you can clearly track configuration changes and their history while reducing manual intervention through automated deployment processes.

For PFCP, we provide Flux as a managed service to implement GitOps on Kubernetes. Flux is an open-source automation tool designed for GitOps on Kubernetes, responsible for monitoring repository changes and automatically applying them to the cluster. Below, we’ll explain how to enable GitOps using Flux.

Installing and Configuring Flux

When referring to “repository” in this documentation, it specifically refers to a GitHub repository.

1. Access the Namespaces page in the portal and create a namespace for using Flux’s Kubernetes resources. For this example, we’ll create a namespace named org-foo--flux.
2. Create a ServiceAccount named flux in the org-foo--flux namespace. Flux will use this ServiceAccount to apply manifests. Note that the ServiceAccount must be named exactly flux; using any other name will result in incorrect operation.

   kubectl create serviceaccount flux --namespace=org-foo--flux
   

3. Grant permissions from the namespaces where you want to apply manifests through the Flux ServiceAccount in the org-foo--flux namespace.

   kubectl create rolebinding flux \
      # The namespace where you want to apply manifests
      --namespace=org-foo--target \
      # The permissions Flux will use for manifest application
      --clusterrole=org-edit \
      # The ServiceAccount Flux will use for manifest application
      --serviceaccount=org-foo--flux:flux
   

   If you have multiple namespaces where you want to apply manifests, you must grant permissions from each respective namespace in a similar manner.

   Warning

   If you want to manage resources requiring org-admin privileges

   Please modify the --clusterrole=org-edit part according to the type of resources you want to manage. For example, if you want to manage RoleBindings with Flux, you’ll need to provide org-admin privileges to the flux ServiceAccount. In such cases, strictly limit access to the org-foo--flux namespace to only organization administrators to prevent unauthorized cluster operations by regular users using the org-admin-privileged flux ServiceAccount.

4. Create the GitRepository and Kustomization resources by writing the following manifest.yaml file. In this example, the manifests located in the main branch of https://github.com/pfcomputing/hello will be applied to the cluster.

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: GitRepository
   metadata:
     name: hello
     namespace: org-foo--flux
   spec:
     interval: 5m
     url: https://github.com/pfcomputing/hello
     ref:
       branch: main
   ---
   apiVersion: kustomize.toolkit.fluxcd.io/v1
   kind: Kustomization
   metadata:
     name: hello
     namespace: org-foo--flux
   spec:
     interval: 10m
     sourceRef:
       kind: GitRepository
       name: hello
     path: "./kustomize"
     prune: true
     timeout: 1m
   

5. Apply the manifests.

   kubectl apply -f manifest.yaml
   

These steps will synchronize the manifest files in your configured repository with the current state of the cluster.

Using Private Git Repositories with Flux

In addition to public repositories, you can also perform GitOps by referencing private repositories. Below are the steps to do so.

1. Create a Secret for accessing the Git repository.

   export KEY_NAME=flux-ssh-key
   ssh-keygen -t ed25519 -f $KEY_NAME
   kubectl create secret generic flux-git-secret \
   --from-literal=known_hosts="$(ssh-keyscan github.com)" \
   --from-file=identity=$KEY_NAME
   

2. Register the output of cat $KEY_NAME.pub as a deployment key with read permissions for the relevant repository on GitHub (GitHub documentation).

3. Modify the GitRepository section in manifest.yaml as follows:

   • Modify the URL to point to your private repository
   • Add a spec.secretRef field with name: flux-git-secret

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: GitRepository
   ...
   spec:
     ...
     url: # Update to your desired private repository URL
     secretRef: # Add this field
       name: flux-git-secret
   

4. Apply the manifests.

   kubectl apply -f manifest.yaml
   

These changes will synchronize the manifests present in your private repository with the current state of the cluster.

Flux also supports using repositories other than GitHub and features notification capabilities. For more detailed information, please refer to the following resources:

• Flux Documentation

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Managing Workload Costs

Warning

The workload cost management feature is currently in preview status. Please note that specifications may change without prior notice, so please use this functionality with caution.

This page explains the features for managing costs associated with your workloads.

Configuring Cost Tags

Warning

The cost aggregation functionality using cost tags will be added at a later date.

For Pod and PersistentVolumeClaim resources created in PFCP’s Kubernetes cluster, you can configure metadata for cost management called “cost tags.” By setting cost tags, you can aggregate cluster usage metrics by each cost tag category.

Cost tags are configured as dedicated labels on resources, with labels allowing up to 16 characters of arbitrary text. If more than 16 characters are specified, only the first 16 characters will be recorded.

• cost.preferred.jp/tag

The following examples demonstrate how to configure cost tags for Pod and PersistentVolumeClaim resources.

apiVersion: v1
kind: Pod
metadata:
  name: jupyter-notebook
  labels:
    # Cost tag configuration
    cost.preferred.jp/tag: dev-research
spec:
  # ...

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jupyter-notebook-data
  labels:
    # Cost tag configuration
    cost.preferred.jp/tag: dev-research
spec:
  # ...

Setting Default Cost Tags for Sub-Namespaces

Cost tags can be configured not only for resources but also for sub-namespaces. When cost tags are set for a sub-namespace, their values will automatically be applied to all Pod and PersistentVolumeClaim resources created under that namespace. Configuring cost tags for sub-namespaces requires organizational administrator privileges.

For example, to configure cost tags for a sub-namespace org-<organization-name>--foo, you would use the following command:

$ kubectl edit subnamespaceanchor org-<organization-name>--foo -n org-<organization-name>

- spec: {}
+ spec:
+   labels:
+   - key: cost.preferred.jp/tag
+     value: dev

If both cost tags are set for the sub-namespace and individual resources, the cost tag value specified for the resource will take precedence. For instance, if the cost tags are configured as shown below, the Pod resource will be processed with the tag dev-research.

• Cost tag value for the sub-namespace: dev
• Cost tag value for the Pod resource: dev-research

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Storage Types and Comparison

PFCP provides two storage options for persistent data storage: file storage and block storage. PFCP’s file storage utilizes NFS (Network File System) as its communication protocol to connect to storage systems, while block storage uses NVMe/TCP.

File Storage

PFCP’s file storage enables data sharing among multiple Pods using the NFS protocol. It is particularly suitable for storing data that is commonly accessed by workloads across an organization, as well as for storing logs and processed results.

Block Storage

PFCP’s block storage leverages NVMe/TCP to enable more advanced operations from Pods against storage systems.

When creating a filesystem on block storage and using it accordingly, multiple Pods cannot access it simultaneously but can perform faster data access. This makes it ideal for use cases requiring high response performance, such as in databases.

When handling block storage as a raw device rather than creating a filesystem, it can be directly used as a standard Linux block device. This approach is useful for achieving functionality not possible with existing filesystems, such as:

• Specialized applications requiring device-level encryption
• Use as virtual disks employing proprietary formats

Comparison

This section outlines the available access modes for the storage types provided by PFCP. The access modes listed in the table are abbreviated as follows: ReadWriteOnce (RWO), ReadOnlyMany (ROX), ReadWriteMany (RWX), and ReadWriteOncePod (RWOP)

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using File Storage

PFCP provides a file storage system that can be mounted by Pods for use. It supports ReadWriteMany file storage, allowing concurrent read/write access from multiple Pods within an organization.

Creating PersistentVolumeClaim and Mounting it to Pods

To utilize the file storage, you must create a PersistentVolumeClaim resource to request the necessary storage space, and then mount the dynamically provisioned PersistentVolume from your Pods. Below is an example demonstrating this process.

1. Create a PersistentVolumeClaim by specifying the organization-specific StorageClass. The available StorageClass name is standard-rwx-<organization-name>.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hello-sample-pvc
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard-rwx-<organization-name>
   

   In this example, we are creating a PersistentVolumeClaim requesting 10GiB of file storage. After applying this manifest to Kubernetes, the system will dynamically create a PersistentVolume, and you can verify that the PersistentVolumeClaim status becomes Bound as follows:

   $ kubectl -n org-<organization-name> get pvc
   NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               VOLUMEATTRIBUTESCLASS   AGE
   hello-sample-pvc   Bound    pvc-c5bb8161-f8f3-4b2d-b001-0aee300b7478   10Gi       RWX            standard-rwx-<organization-name>   <unset>   3d2h
   

2. You can now use this PersistentVolumeClaim by mounting it in your Pods to access file storage.

   apiVersion: v1
   kind: Pod
   metadata:
     name: jupyter-notebook
   spec:
     containers:
     - name: jupyter-notebook
       image: quay.io/jupyter/scipy-notebook:2024-03-14
       volumeMounts:
       - mountPath: "/hello-sample"
         name: hello-sample-pv
     volumes:
     - name: hello-sample-pv
       persistentVolumeClaim:
         claimName: hello-sample-pvc
   

Limitations

All File Operations Are Processed as Access by the nobody User

In PFCP’s file storage system, all operations on files and directories within the volume are processed as performed by the nobody user (UID 65534, GID 65534). This applies uniformly to all Linux users, including the root user (UID 0).

Note

This limitation only applies to file storage systems. It does not exist for block storage.

As an example, below shows a user with the ubuntu account (UID 1000) creating a new file on the volume, though the operation is processed as access by the nobody user. You can verify that the created file has nobody:nogroup as its owner and group (65534:65534).

ubuntu@pv-test:~$ id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
ubuntu@pv-test:~$ ls -al /data-rwx
total 8
drwxrwxrwx 3 nobody nogroup 4096 Nov 10 23:28 .
drwxr-xr-x 1 root   root    4096 Nov 10 23:09 ..
ubuntu@pv-test:~$ touch /data-rwx/testfile
ubuntu@pv-test:~$ ls -al /data-rwx/testfile
-rw-r--r-- 1 nobody nogroup 0 Nov 10 23:39 /data-rwx/testfile

Even when performing update operations on the /data-rwx/testfile file, which has nobody as its owner, the operation is processed as by the nobody user and succeeds without permission errors.

ubuntu@pv-test:~$ ls -al /data-rwx/testfile
-rw-rw-r-- 1 nobody nogroup 0 Nov 10 23:39 /data-rwx/testfile
ubuntu@pv-test:~$ date >/data-rwx/testfile
ubuntu@pv-test:~$ cat /data-rwx/testfile
Mon Nov 10 23:40:20 UTC 2025
ubuntu@pv-test:~$ ls -al /data-rwx/testfile
-rw-rw-r-- 1 nobody nogroup 29 Nov 10 23:40 /data-rwx/testfile

However, due to this limitation, certain operations on file storage are unavailable, including chown/chgrp commands to modify file owners/groups, and the Pod securityContext fsGroup feature that changes the volume’s group during Pod execution.

chown/chgrp: Results in Operation not permitted errors

When attempting to execute chown/chgrp commands to modify file owners/groups on the volume, an Operation not permitted error occurs and the operation fails as shown below.

root@pv-test:~# id
uid=0(root) gid=0(root) groups=0(root)
root@pv-test:~# chown root /data-rwx/testfile
chown: changing ownership of '/data-rwx/testfile': Operation not permitted
root@pv-test:~# chgrp root /data-rwx/testfile
chgrp: changing group of '/data-rwx/testfile': Operation not permitted

The same error occurs when executing kubectl cp or tar/rsync commands. This is because these commands internally modify file/directory owners. You can avoid errors by using options that prevent owner/group changes with each command.

• kubectl cp: Use the --no-preserve option
• tar: Use the --no-same-owner option
• rsync: Use the --no-owner option

Pod securityContext fsGroup: Configuration is ignored and Pod runs as configured

The Pod SecurityContext fsGroup feature, which allows changing the volume’s group during Pod execution, is ignored for file storage volumes and the Pod will run as configured.

References

• Persistent Volumes | Kubernetes

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using Block Storage

PFCP supports block storage as PersistentVolume resources with either ReadWriteOnce or ReadWriteOncePod access modes. For ReadWriteOncePod volumes, access is restricted to a single pod only.

Creating and Mounting a Filesystem from PersistentVolumeClaim to a Pod

To utilize block storage by creating a filesystem on it, you should first create a PersistentVolumeClaim resource to request the necessary storage capacity, then mount the dynamically provisioned PersistentVolume from within your pod. Below is an example demonstrating this process.

1. Create a PersistentVolumeClaim by specifying the organization-specific StorageClass. The available StorageClass name is standard-rwo-<organization-name>.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hello-sample-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard-rwo-<organization-name>
     volumeMode: Filesystem
   

   In this example, we’re creating a PersistentVolumeClaim to request a filesystem on 10GiB of block storage. Since volumeMode defaults to Filesystem, this specification is technically optional—the resulting volume would be created identically without this explicit declaration. After applying this manifest to Kubernetes, the PersistentVolume will be created dynamically, and you can verify that the PersistentVolumeClaim status becomes Bound as follows:

   $ kubectl -n org-<organization-name> get pvc
   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           VOLUMEATTRIBUTESCLASS   AGE
   hello-sample-pvc   Bound    pvc-ac98a6ff-58cd-4bef-8057-4837949107d0   10Gi       RWO            standard-rwo-<organization-name>   <unset>                 7s
   

2. By specifying this PersistentVolumeClaim in your pod configuration, you can mount it and use it as file storage.

   apiVersion: v1
   kind: Pod
   metadata:
     name: jupyter-notebook
   spec:
     containers:
     - name: jupyter-notebook
       image: quay.io/jupyter/scipy-notebook:2024-03-14
       volumeMounts:
       - mountPath: "/hello-sample"
         name: hello-sample-pv
     volumes:
     - name: hello-sample-pv
       persistentVolumeClaim:
         claimName: hello-sample-pvc
   

Using Block Storage Directly from a Pod Without Creating a Filesystem

To use block storage directly as a raw block device, you should create a PersistentVolumeClaim to request the necessary storage capacity, then access the dynamically provisioned PersistentVolume through your pod via a device file. Below is an example demonstrating this approach.

1. Create a PersistentVolumeClaim by specifying the organization-specific StorageClass. The available StorageClass name is standard-rwo-<organization-name>.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: hello-sample-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard-rwo-<organization-name>
     volumeMode: Block
   

   In this example, we’re creating a PersistentVolumeClaim to use 10GiB of block storage as a raw block device. By specifying Block as the volumeMode, we prevent any filesystem creation operations on the block storage. After applying this manifest to Kubernetes, the PersistentVolume will be created dynamically, and you can verify that the PersistentVolumeClaim status becomes Bound as follows:

   $ kubectl -n org-<organization-name> get pvc
   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           VOLUMEATTRIBUTESCLASS   AGE
   hello-sample-pvc   Bound    pvc-fb34c017-ce51-488c-a445-65981b031e0b   10Gi       RWO            standard-rwo-<organization-name>   <unset>                 119s
   

2. By specifying this PersistentVolumeClaim in your pod configuration as a device file, you can access it directly as a block device from within your pod.

   apiVersion: v1
   kind: Pod
   metadata:
     name: block-demo
   spec:
     containers:
     - name: block-demo
       image: ubuntu
       command:
       - sleep
       - "3600"
       volumeDevices:
       - name: hello-sample-pv
         devicePath: /dev/block
     volumes:
     - name: hello-sample-pv
       persistentVolumeClaim:
         claimName: hello-sample-pvc
   

   This example demonstrates how to operate the block device from within the pod. Here, the device file is mapped to nvme2n1, and you can confirm it is recognized as a block device with the specified 10GiB capacity from the PersistentVolumeClaim.

   $ kubectl -n org-<organization-name> exec -it block-demo -- bash
   root@block-demo:/# ls -lF /dev/block
   brw-rw---- 1 root disk 259, 20 Aug  9 03:49 /dev/block
   root@block-demo:/# lsblk /dev/block
   NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
   nvme2n1 259:20   0  10G  0 disk
   root@block-demo:/#
   

References

• Persistent Volumes | Kubernetes

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Sharing File Storage Across Namespaces

Among the two types of persistent storage provided by PFCP, file storage can be simultaneously read from and written to by multiple Pods across different namespaces within the same organization.

Sharing File Storage

Traditional Kubernetes file storage is restricted to read/write operations only within the namespace where the PersistentVolumeClaim (PVC) resource is created. PFCP offers a feature¹ that allows file storage to be shared across namespaces, enabling read/write access even from namespaces different from the one where the PVC was created, provided they are within the same organization.

Below we explain how to share file storage across namespaces within the same organization.

Step 1. Create a PVC in the source namespace

For illustration purposes, let’s assume:

• The PVC to be created is named pvc1
• The namespace providing the file storage is org-example--namespace1
• The namespace that will consume the storage provided by pvc1 is org-example--namespace2 First, in the namespace providing the file storage, apply the trident.netapp.io/shareToNamespace annotation to the PVC you wish to share. This allows sharing access from the specified namespace.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
  namespace: org-example--namespace1
  annotations:
    trident.netapp.io/shareToNamespace: org-example--namespace2
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: standard-rwx-example
  resources:
    requests:
      storage: 100Gi

Tip

• Multiple target namespaces can be specified by separating them with commas. Example: trident.netapp.io/shareToNamespace: org-example--namespace2,org-example--namespace3,org-example--namespace4
• Specifying an asterisk * allows access from any namespace. Example: trident.netapp.io/shareToNamespace: *
• The trident.netapp.io/shareToNamespace annotation for a PVC can be added or modified at any time².

Step 2. Create a TridentVolumeReference in the target namespace

In the namespace where the file storage will be shared, create a custom resource TridentVolumeReference. This configuration informs the system which PVC in which namespace you wish to reference.

In this example, we want to reference the PVC pvc1 shared from namespace org-example--namespace1 from namespace org-example--namespace2, so the configuration would look like this:

apiVersion: trident.netapp.io/v1
kind: TridentVolumeReference
metadata:
  name: my-first-tvr
  namespace: org-example--namespace2
spec:
  pvcName: pvc1
  pvcNamespace: org-exapmle--namespace1

Step 3. Create a PVC in the target namespace

In the namespace where the file storage will be shared, create a PVC. By applying the trident.netapp.io/shareFromPVC annotation, you specify which PVC from which namespace to use.

In this example, we want to share the PVC pvc1 located in namespace org-example--namespace1, so the configuration would look like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    trident.netapp.io/shareFromPVC: org-example--namespace1/pvc1
  name: pvc2
  namespace: org-example--namespace2
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: standard-rwx-example
  resources:
    requests:
      storage: 100Gi

Note

The storage size specified in the target PVC cannot be larger than the size of the source PVC.

Step 4. Use it just like a regular PVC

Mount the PVC into a Pod and use it as file storage in exactly the same way as with a regular PVC.

Warning

The target PVC will consume quota resources

The target PVC will be treated as having been allocated resource.requests.storage, just like a regular PVC. This means if ResourceQuotas are set in the organization’s namespaces, creating the target PVC will consume those quotas. Since the target PVC’s request values can be different from the source PVC’s settings, you can minimize ResourceQuota consumption by setting resource.requests.quota to as small a value as possible, such as 1 (1 byte). However, you cannot request a value larger than the source PVC’s settings.

Deleting Shared File Storage

Even when file storage is shared across multiple namespaces, there is no special requirement for deletion order. Simply remove the PVC from the target namespace in the usual manner. The system will automatically delete the volume when it detects that the shared file storage is no longer referenced from any namespace.

Related Links

• Share an NFS volume across namespaces | Astra Trident

1. This feature utilizes the TridentVolumeReference from Astra Trident. ↩

2. The actual evaluation of sharing access permissions occurs when creating the sharing-side PVC as described below. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Persistent Storage Snapshots

Persistent storage snapshots provide a convenient mechanism for backing up and restoring data. They allow you to create copies of volumes at specific points in time, enabling you to revert volumes to previous states or create new volumes. Once created, snapshots maintain their state even if the underlying volume is updated. This makes them ideal for use cases such as backups and versioning.

This documentation explains how to create, manage, and utilize snapshots.

Creating a Snapshot from a Volume

You create a snapshot of the current state of an existing volume.

Suppose you have the following PersistentVolumeClaim (PVC): You want to create a snapshot for this PVC named hello-sample-pvc.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hello-sample-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Mi
  storageClassName: standard-rwo-<organization-name>
  volumeMode: Filesystem

1. Create a VolumeSnapshot resource manifest specifying the name of the PVC from which to create the snapshot.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: hello-snapshot-v1
spec:
  source:
    persistentVolumeClaimName: hello-sample-pvc

Applying this manifest to Kubernetes will save the volume’s state at that moment, creating a new snapshot.

2. You can verify whether the snapshot was created successfully using kubectl get volumesnapshots.

$ kubectl get volumesnapshots
NAME               READYTOUSE   SOURCEPVC       SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                      CREATIONTIME   AGE
hello-snapshot-v1  true         hello-sample-pvc                           ...Mi         trident         snapcontent-..........   ..d            ..d

A successfully created snapshot will have the READYTOUSE field set to true. In this case, the snapshot from the PersistentVolumeClaim was created successfully.

Note

If READYTOUSE is not true, the creation process is not yet complete. Creating snapshots may take several seconds to a minute as data is being copied.

Restoring a Volume from a Snapshot

You can create a new volume from a snapshot by writing a PVC manifest as shown below.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hello-sample-pvc-restored
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 20Mi
  storageClassName: standard-rwo-<organization-name>
  dataSource:
    name: hello-snapshot-v1
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Compute Node Types and Comparison

PFCP offers two types of compute nodes:

• Dedicated Nodes: Compute nodes reserved exclusively for your organization.
• Shared Nodes: Compute nodes shared among multiple organizations.

Below is a comparison of feature support for each type of compute node.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using Dedicated Nodes

Dedicated nodes are a specific type of compute node that can be reserved for your organization’s workloads. Pods created within your organization’s Namespace will be automatically scheduled on dedicated nodes unless explicitly specified to run on shared nodes see Shared Nodes. This eliminates the need for users to manually select nodes (by specifying a nodeSelector).

Dedicated nodes allocated to your organization can be accessed through the ReservedNode resource. The following command example lists all dedicated nodes assigned to your organization:

kubectl get reservednodes

The ReservedNode resource can be managed similarly to Kubernetes’ Node resource, allowing selection using either Label Selectors or Field Selectors. The fields supported by Field Selectors are as follows:

• metadata.name
• spec.unschedulable

Important note: The ReservedNode resource is read-only and cannot be modified.

Warning

Impact on Workloads During Maintenance

Be aware that maintenance activities involving compute node reboots may temporarily render some dedicated nodes unavailable.

• For organizations with fewer than 4 dedicated node reservations, one node will be temporarily unavailable at a time. For 4 or more reservations, up to 25% (rounded down) of nodes may become temporarily unavailable.
• Organizations with only one dedicated node reservation will experience complete node unavailability during maintenance.

For details on how maintenance activities may affect your workloads, please refer to the Maintenance Policy.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using Shared Nodes

Warning

This feature is currently only available in Service Release 1 (SR1). There is a limited number of shared nodes available, and their availability may vary depending on usage conditions. Visualization of the shared node usage status is planned for future implementation.

PFCP provides nodes shared among multiple organizations. Unlike dedicated nodes allocated to individual organizations, shared nodes are used by multiple organizations simultaneously.

Available Nodes

The following nodes are provided as shared resources:

• MN-Server 2 (preferred.jp/mncore2)

Usage Instructions

By specifying a dedicated PriorityClass for your Pod, it will be automatically scheduled to shared nodes. While PriorityClass primarily functions for setting Pod priority, it also serves to control which type of node the Pod should be scheduled on. Therefore, users do not need to manually specify node selection (via nodeSelector). Below is an example of how to use a Pod with shared nodes:

apiVersion: v1
kind: Pod
metadata:
  name: shared
spec:
  priorityClassName: shared-standard # or shared-best-effort
  ...(omitted)...

The PriorityClasses available for using shared nodes are as follows:

• shared-standard: Has the highest priority among shared node PriorityClasses and is not subject to preemption (cannot be forcibly removed)
• shared-best-effort: May be preempted by Pods using the shared-standard PriorityClass

For detailed information on Pod priority and preemption, please refer to the official Kubernetes documentation.

Each organization has a resource quota for MN-Core 2 usage on shared nodes, with limits set for each PriorityClass. You can check the available MN-Core 2 quotas using the following command:

kubectl -n org-<organization-name> get hrq shared-standard shared-best-effort -o yaml

Constraints

• There are limits on the maximum resource allocation per MN-Core 2. When requesting two MN-Core 2 resources per Pod, the quota is doubled. The maximum limits per MN-Core 2 for each resource are as follows:

  Resource	Maximum per MN-Core 2
  CPU	7000m
  Memory	125Gi
  Ephemeral Storage	80Gi

Security

Unlike dedicated nodes, shared nodes share the same kernel with multiple organizations’ workloads running on the same physical machine. For stronger security isolation between different organizations, we recommend using dedicated nodes instead.

Additional Security Measures

Shared nodes employ Linux User Namespaces to enforce separation between container UID/GIDs in the Pod and the host system. This technology reduces the risk by preventing attackers who gain access from a container to the host from operating as root (user with UID 0) within the host environment. Linux User Namespaces are known to mitigate numerous past container-related CVEs¹. For detailed information, please refer to the official Kubernetes User Namespaces documentation.

1. https://github.com/kubernetes/enhancements/tree/217d790720c5aef09b8bd4d6ca96284a0affe6c2/keps/sig-node/127-user-namespaces#motivation ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using User-Managed Container Image Repositories

Note

This page provides instructions for using container images stored in user-managed container image registries. For information on using container images provided by PFCP, please refer to About PFCP-Provided Container Images.

PFCP supports pulling container images from private registries hosted on Amazon ECR and Google Artifact Registry. A system called the Image pull secrets provisioner is deployed, which automatically creates and updates secrets required for pulling images from ECR and Google Artifact Registry.

Launching Workloads Using Container Images from ECR

1. On AWS, configure identity federation for the Kubernetes ServiceAccount you intend to use.

   • Configure trust relationships to enable Pods using this ServiceAccount to pull images from ECR.
   • For configuration details, refer to the AWS documentation Create an OpenID Connect (OIDC) identity provider in IAM - AWS Identity and Access Management.
   • For the provider (issuer) URL, please refer to Regions and Compute Nodes.¹
   • If you’re using Terraform for AWS manifest management, also see the Terraform configuration example.

2. Apply the following annotations to the ServiceAccount you wish to use:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     namespace: NAMESPACE
     name: SERVICE-ACCOUNT-NAME
     annotations:
       # Specify the ECR registry containing the container images you want to use.
       imagepullsecrets.preferred.jp/registry: 999999999999.dkr.ecr.LOCATION.amazonaws.com
       # Specify the aud value used for ECR access via identity federation.
       imagepullsecrets.preferred.jp/audience: sts.amazonaws.com
       # Specify the IAM Role to be used for AssumeRole during ECR access.
       imagepullsecrets.preferred.jp/aws-role-arn: arn:aws:iam::999999999999:role/ROLE-NAME
   

3. Set the Pod to use this ServiceAccount by specifying the .spec.serviceAccountName field:

   apiVersion: v1
   kind: Pod
   metadata:
     name: POD-NAME
   spec:
     serviceAccountName: SERVICE-ACCOUNT-NAME
     ...
   

4. Launch the Pod and verify that it can successfully pull images from ECR.

Launching Workloads Using Container Images from Google Artifact Registry

1. On Google Cloud, configure identity federation for the Kubernetes ServiceAccount you intend to use.

   • Configure trust relationships to enable Pods using this ServiceAccount to pull images from Google Artifact Registry.
   • For configuration details, refer to the Google Cloud documentation Configure workload identity federation with Kubernetes | IAM Documentation | Google Cloud.
   • For the provider (issuer) URL, please refer to Regions and Compute Nodes.¹
   • If you’re using Terraform for Google Cloud manifest management, also see the Terraform configuration example.

2. Apply the following annotations to the ServiceAccount you wish to use:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     namespace: NAMESPACE
     name: SERVICE-ACCOUNT-NAME
     annotations:
       # Specify the Google Artifact Registry containing the container images you want to use.
       imagepullsecrets.preferred.jp/registry: LOCATION-docker.pkg.dev
       # Specify the aud value used for Google Artifact Registry access via identity federation.
       imagepullsecrets.preferred.jp/audience: //iam.googleapis.com/projects/999999999999/locations/global/workloadIdentityPools/POOL-NAME/providers/PROVIDER-NAME
       # Specify the resource name of the Workload Identity provider for identity federation.
       imagepullsecrets.preferred.jp/googlecloud-workload-identity-provider: projects/999999999999/locations/global/workloadIdentityPools/POOL-NAME/providers/PROVIDER-NAME
       # Specify the email address of the Google service account used for identity federation.
       imagepullsecrets.preferred.jp/googlecloud-service-account-email: SERVICE-ACCOUNT-ID@PROJECT-NAME.iam.gserviceaccount.com
   

3. Set the Pod to use this ServiceAccount by specifying the .spec.serviceAccountName field:

   apiVersion: v1
   kind: Pod
   metadata:
     name: POD-NAME
   spec:
     serviceAccountName: SERVICE-ACCOUNT-NAME
     ...
   

4. Launch the Pod and verify that it can successfully pull images from Google Artifact Registry.

For more detailed information, please refer to the README for the image pull secrets provisioner.

1. If you are using multiple PFCP clusters, configuration must be performed separately for each cluster. ↩ ↩2

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Using Container Images Provided by PFCP

This page provides instructions for utilizing the container images that PFCP provides as part of its cloud service. For information on using container images managed by your organization, please refer to About Customer-Managed Container Images.

Warning

PFCP-provided container images can only be accessed from within the PFCP Kubernetes cluster. Attempting to retrieve these images from any other environment will fail.

List of PFCP-Provided Container Images

You can view the list of container images provided by PFCP on the Images page of the portal.

Launching Workloads Using PFCP-Provided Container Images

Follow these steps to launch workloads using PFCP-provided container images:

1. Specify the desired container image in the Pod’s spec.containers.image field:

   apiVersion: v1
   kind: Pod
   metadata:
     name: POD-NAME
   spec:
     containers:
     - name: CONTAINER-NAME
       image: registry.pfcomputing.internal/IMAGE:TAG
     ...
   

2. Launch the Pod and verify that you can successfully retrieve the container image.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Creating Interactive Work Environments

To create an interactive work environment on a cluster, use the Workspace feature. Workspaces are interactive work environments accessible via web browsers. They allow users to utilize PFCP’s computational resources through interfaces like JupyterLab.

Workspace Isolation

There are two isolation units for individual workspace instances: individual users and namespaces. When creating a workspace, you can select either isolation type during creation.

Individual User Isolation

Workspaces isolated at the individual user level are accessible only by the user who created the workspace. This makes them ideal for securely storing personal credentials such as SSH private keys.

Other users, including organizational administrators, cannot access these workspaces. However, organizational administrators can suspend or delete workspaces.

Below is a comprehensive list of available operations:

Namespace Isolation

Workspaces isolated at the namespace level are accessible by users with the [org-edit Role] for the namespace in which the workspace was created. This enables sharing workspaces among users who have appropriate permissions for the namespace.

Users without the org-edit Role for the namespace cannot access these workspaces.

[org-edit Role]: ./role-binding.md#pfcp-Standard-Provided Roles and Groups

Creating a Workspace

Note

Workspaces utilize computational resources from your organization. If computational resources are insufficient, workspace creation may fail.

Note

Workspaces isolated at the individual user level can only be created in the root namespace. Therefore, creation requires granting either the org-edit or org-workspace-edit ClusterRole for the root namespace. For instructions on assigning ClusterRoles, see Creating a RoleBinding.

1. Navigate to the Workspaces page in the portal and click the Create New button.
2. Fill out the form and click the Create button.

Accessing a Workspace

1. Access the Workspaces page in the portal.

2. Click on the URL link in the column corresponding to the workspace you wish to access.

   Pausing and Resuming Workspaces

Warning

Pausing a workspace will delete all data contained within it. Data stored in PersistentVolumes will be preserved during the pause period.

Pausing unused workspaces can help conserve computational resources.

1. Access the Workspaces page in the portal.

2. Select the workspace you want to pause and click the Pause button. Pause-enabled workspaces can be resumed using the same procedure.

3. Access the Workspaces page in the portal.

4. Select the workspace you want to resume and click the Resume button.

Deleting Workspaces

Warning

Deleting an entire workspace will permanently remove all data contained within it, as well as any PersistentVolumes created from that workspace.

1. Access the Workspaces page in the portal.
2. Select the workspace you wish to delete and click the Delete button.

Managing Workspaces Using Kubernetes Manifests

In addition to managing workspaces through the portal, you can also administer them using Kubernetes manifests. Using manifests enables automated workspace management and ensures consistent reproduction of workspace configurations.

Workspace Custom Resource

Each workspace instance is represented by a Workspace custom resource.

The Workspace resource automatically creates the underlying Pod that constitutes the workspace. You can manage workspaces by creating, updating, or deleting Workspace resources.

Workspace resources are defined in the following format:

apiVersion: preferred.jp/v1alpha1
kind: Workspace
metadata:
  name: ...
  namespace: ...
spec:
  owner:
    type: Individual
  presetRef: ...
  podTemplate: ...
  volumeClaimTemplates:
  - ...

Tip

You can also view detailed field descriptions by running the kubectl explain workspace command.

• spec.owner.type field

  • Specifies the isolation type of the workspace.
  • Use Individual for user-specific isolation or Namespace for namespace-based isolation.

• spec.presetRef field

  • Specifies the preset to be applied to the workspace.

• spec.podTemplate field

  • Specifies the PodTemplateSpec to be applied to the underlying Pod that forms the workspace.

• spec.volumeClaimTemplates field

  • Specifies a list of [PersistentVolumeClaims] to be created from the workspace.
  • By referencing the PersistentVolumeClaims created in spec.podTemplate, the workspace can utilize PersistentVolumes.

Presets

You can pre-define the configuration for the underlying Pod of a workspace as a preset. Defining presets simplifies the management of workspaces with similar configurations.

A preset specifies default values for the spec.podTemplate field that will be applied to the workspace. When creating a workspace’s underlying Pod from a Workspace resource, any fields not specified in the Workspace resource’s spec.podTemplate field will be inherited from the preset. Fields explicitly defined in the spec.podTemplate field will override the preset values and will be used to create the Pod.

There are two types of presets: ClusterWorkspacePreset custom resources and WorkspacePreset custom resources.

ClusterWorkspacePreset Custom Resource

A ClusterWorkspacePreset custom resource is a shared preset used across the entire PFCP-managed cluster. It can be utilized by workspaces across all organizations.

Available ClusterWorkspacePreset resources can be listed using the kubectl get clusterworkspacepreset command (or kubectl get cwspreset). To apply a specific ClusterWorkspacePreset, you can set the spec.presetRef field in your Workspace resource as follows:

apiVersion: preferred.jp/v1alpha1
kind: Workspace
spec:
  presetRef:
    apiVersion: preferred.jp/v1alpha1
    kind: ClusterWorkspacePreset
    name: NAME

If the spec.presetRef field is omitted, the default ClusterWorkspacePreset will be applied. Below is a partial excerpt of some values from the default ClusterWorkspacePreset:

apiVersion: preferred.jp/v1alpha1
kind: ClusterWorkspacePreset
metadata:
  name: default
spec:
  podTemplate:
    spec:
      containers:
      - name: workspace
        image: registry.pfcomputing.internal/mncore-sdk/mncore-sdk-full
        command:
        - /app/jupyter/bin/jupyter
        - lab

If the spec.podTemplate field is not specified in the Workspace resource, a Pod will be created with a workspace container that launches JupyterLab using the MN-Core SDK container image according to these values. You can override this default configuration by specifying container images, commands, and other parameters in the spec.podTemplate field. Additionally, you can configure resource requests for the workspace container and add other containers in addition to the workspace container.

WorkspacePreset Custom Resource

The WorkspacePreset custom resource is a shared preset defined at the namespace level. Users with the [org-edit Role] for the namespace can create, update, or delete this resource.

WorkspacePreset resources within a namespace can be listed using the kubectl get workspacepreset command (or kubectl get wspreset). You can apply a specific WorkspacePreset by setting the spec.presetRef field in the Workspace resource as follows:

apiVersion: preferred.jp/v1alpha1
kind: Workspace
spec:
  presetRef:
    apiVersion: preferred.jp/v1alpha1
    kind: WorkspacePreset
    name: NAME

Note

For a Workspace to reference a WorkspacePreset, both the Workspace and WorkspacePreset must reside in the same namespace.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Connecting to Your Work Environment via Visual Studio Code

You can connect to your work environment created using the Workspace feature from Visual Studio Code (VSCode). You can perform code editing and debugging using your locally configured VSCode environment.

Connecting Through a Tunnel

This method uses VSCode’s Remote Tunnels feature to connect to your workspace.

Note

To use this method, you must agree to the Visual Studio Code Server License Terms and Microsoft Privacy Statement.

Warning

This method establishes a connection to your workspace via a route that does not apply PFCP’s authentication and authorization mechanisms. Therefore, even if you’ve integrated PFCP with an authentication service, note that service usage will occur without going through that authentication infrastructure. Additionally, managing and terminating Remote Tunnels connections for individual users is not coordinated with PFCP and must be handled by the user themselves.

Preparation in Your Workspace

1. Create a workspace.
   • Refer to Creating a Workspace for details.
2. Open a JupyterLab terminal and execute the following command to start a tunnel:
   • The default container image used in your workspace includes the code command. If you plan to use an alternative image, ensure the code command is installed.

   code tunnel
   

3. Follow the instructions provided by the command to complete initial setup.

Connecting from VSCode

1. Launch your locally installed VSCode.
2. If the Remote - Tunnels extension is not already installed, install it via the Remote - Tunnels extension.
3. From the command palette, select Remote-Tunnels: Connect to Tunnel… to begin the connection process.
4. Follow the prompts to complete authentication and connect to your workspace.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Creating a Distributed Batch Job

This page explains how to create a ParallelJob for executing distributed batch processing on a PFCP cluster.

Note

As this is an experimental feature, the API schema may change in the future.

Overview

A ParallelJob is a Kubernetes custom resource designed to simplify the deployment and management of distributed batch jobs that coordinate across multiple nodes. ParallelJobs offer the following key features:

• Support for multiple frameworks: Supports distributed processing frameworks such as MPI.
• Gang Scheduling: Ensures all Pods within a distributed batch job are scheduled simultaneously.
• Job failure retries: Automatically retries the entire ParallelJob if any single job fails.

Runtimes Supported by ParallelJob

Note

While we plan to support multiple runtimes in the future, currently only the Open MPI runtime is available.

Open MPI

Supports parallel computing using the Open MPI implementation of the Message Passing Interface (MPI). In an Open MPI job, a single launcher Pod connects to multiple worker Pods via SSH to execute parallel computations using MPI. The launcher Pod also functions as a worker with rank=0.

Below is an example configuration for a ParallelJob using the MPI runtime:

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  # Specifies the MPI runtime to use
  runtimeRef:
    name: mpi-openmpi

  # Number of Pods including both launcher and worker Pods.
  # For MPI runtimes, there is one launcher Pod and the remainder are worker Pods.
  numPods: 3
  # Number of MPI processes per Pod, which corresponds to the "slots" in the MPI hostfile.
  # For accelerator-based computations, this should specify the number of accelerators per Pod; for flat MPI, the number of CPU cores; and for hybrid parallelism, typically set to 1.
  numProcPerPod: 2

  # Configuration for the launcher Pod template. Specified in PodTemplateSpec format.
  # If the `main` container is missing, the ParallelJob creation will fail.
  launcher:
    spec:
      containers:
      - name: main # The container running MPI should be named `main`.
        image: ghcr.io/pfnet/parallel-controller/openmpi:v0.1.0
        command:
        - sh
        - -c
        - |
          mpirun --allow-run-as-root sh -c '
            cat > hello_mpi.c << "EOF"
            #include <mpi.h>
            #include <stdio.h>

            int main(int argc, char *argv[]) {
              MPI_Init(&argc, &argv);
              int rank, world_size;
              MPI_Comm_rank(MPI_COMM_WORLD, &rank);
              MPI_Comm_size(MPI_COMM_WORLD, &world_size);
              printf("Hello from MPI process %d rank in %d processes\n", rank, world_size);
              MPI_Finalize();
              return 0;
            }
            EOF
            mpicc -o hello_mpi hello_mpi.c
            ./hello_mpi'

  # (Optional) Configuration for the worker Pod template. Specified in PodTemplateSpec format.
  # Worker settings inherit from the launcher, and an automatically generated container for communication with the launcher will be named `main`.
  # worker: nil

Note

Using custom container images

When using custom container images with the Open MPI runtime, the following software must be included:

• Open MPI
• ssh (SSH client)
• sshd (SSH server)

Example Configurations for Different Scenarios

Below are example configurations commonly used in practical deployment scenarios.

Retry Configuration for Job Failures

By default, if any job within a ParallelJob fails, the entire ParallelJob will be retried up to a maximum of 3 times. To change the retry count, set the .spec.failurePolicy.maxRestarts field. Below is an example setting the retry count to 1.

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: mpi-sample-job
spec:
  failurePolicy:
    # Maximum number of times to rerun the entire ParallelJob on job failure. (Default: 3)
    maxRestarts: 1
...

Note

Job interruptions caused by preemption or eviction are not counted towards the retry limit.

Specifying GPU/RDMA Resources

When using GPU or RDMA resources in a distributed job, specify the corresponding resource requirements in the Pod template. While NCCL and UCX configuration files are automatically generated, they must be explicitly sourced. Below is an example requesting GPU and RDMA devices for the launcher Pod and loading NCCL and UCX configuration files (this will be automated in future updates).

apiVersion: preferred.jp/v1alpha1
kind: ParallelJob
metadata:
  name: sample-job
spec:
  launcher:
    spec:
      containers:
      - name: main
        command: ["/bin/sh", "-c"]
        args:
        # Note: While NCCL and UCX configuration files are automatically generated, they must be explicitly sourced.
        - |
          [ -f "$RDMA_NCCL_CONF" ] && . "$RDMA_NCCL_CONF"
          [ -f "$RDMA_UCX_CONF" ] && . "$RDMA_UCX_CONF"

          mpirun --allow-run-as-root something-using-gpu-rdma
        resources:
          limits:
            nvidia.com/gpu: "2"
            preferred.jp/rdma: "1"
...

Note

RDMA resources are only available on the following clusters:

• IK1-01
• YH1-01

Troubleshooting

When encountering issues during distributed job execution, follow these steps to troubleshoot:

Checking Job Status

To check the status of a ParallelJob, execute the following command:

# Check ParallelJob status
kubectl get paralleljobs sample-job

# View detailed information including events
kubectl describe paralleljobs sample-job

To check the status of related Job/Pods, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Check the status of related Jobs/Pods
kubectl get jobs,pods -l jobset.sigs.k8s.io/jobset-name=${jobset_name}

Checking Pod Logs

To retrieve logs from the rank=0 Pod, execute the following command:

# Set the ParallelJob's associated JobSet name to a variable
jobset_name=$(kubectl get paralleljobs sample-job -o jsonpath={".status.jobGroupName"})

# Display logs
kubectl logs -f $(kubectl get pod -l jobset.sigs.k8s.io/jobset-name=${jobset_name},jobset.sigs.k8s.io/job-global-index=0 -o name)

Note

Logs can only be checked if the Pod exists. For successful completion, all Pods remain; for error termination, only the errored Pod remains. Additionally, deleting a ParallelJob will also remove all associated Pods.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Exposing Workloads as Web Applications

The WebApp Identity-Aware Proxy (WebApp IAP) feature in PFCP allows you to expose your workloads as web applications to the internet. The exposed web application will automatically implement authentication for access, allowing only users belonging to the same organization to access it via their browsers.

This section explains how to expose your web application to the internet using WebApp IAP.

Note

For exposing your workloads as web APIs accessible via CLI or other interfaces, please refer to Exposing Workloads as Web APIs.

Exposing a Web Application to Your Entire Organization

1. Prepare the workload you want to expose and its corresponding Service resource. For this example, assume you can access the workload by connecting to port 80 of the example-svc Service.

2. Create an Ingress manifest using the following template:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-ingress
   spec:
     rules:
     - # Specifies the domain to assign to the Ingress.
       # Note: Unlike web API exposure, you must specify a subdomain of `ingress.pfcomputing.com`.
       host: example.<organization-name>.<cluster-name>.ingress.pfcomputing.com
       http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             # Specifies the name of the Service and its port to expose.
             service:
               name: example-svc
               port:
                 number: 80
   

   Warning

   Subdomain Restrictions

   For web application Ingresses, only the domain *.<organization-name>.<cluster-name>.ingress.pfcomputing.com can be used.

   For example, if your organization name is foo and your cluster name is sr1-01, the valid subdomain would be *.foo.sr1-01.ingress.pfcomputing.com.

3. Access the Public Endpoints page in the portal, select your cluster name and namespace, and verify that the Ingress’s subdomain appears in the list of generated public endpoints.

4. Access the specified subdomain in your browser, log in ¹, and then verify that you can access the exposed service.

Exposing a Web Application to Select Users Within Your Organization

You can restrict the exposure of your web application to specific users or user groups within your organization.

First, follow the “Exposing a Web Application to Your Entire Organization” instructions to create the Ingress. Then, for the Ingress you want to restrict access to, add the following annotations. The value should be a comma-separated list of email addresses or user group names for users you wish to allow access.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    # To allow specific users, add the allowed-users annotation
    ingress.preferred.jp/allowed-users: "foo@example.com, bar@example.com, baz@example.com"
...

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    # To allow specific user groups, add the allowed-groups annotation
    ingress.preferred.jp/allowed-groups: "foo-group, bar-group"
...

If you add both annotations, access will be permitted for both specified users and user groups.

For managing user groups, please refer to Managing Organizational Users.

Tip

You can check which user groups you belong to using the kubectl auth whoami command. If you’re a member of the bar user group in organization foo, the Groups list will display oidc:org-foo/bar.

Verifying Access Restrictions

After applying the annotations, actually access the Ingress to verify that access restrictions are functioning correctly.

Warning

Changes to user groups may take time to propagate to existing logged-in user sessions. Try logging out and back in, or clearing your browser cookies.

When access restrictions are enabled, the Ingress will automatically include the nginx.ingress.kubernetes.io/auth-url annotation. If this annotation is missing, review your manifest for any annotation typos.

Restoring Exposure to Your Entire Organization

Remove all the following annotations from the Ingress:

• ingress.preferred.jp/allowed-users
• ingress.preferred.jp/allowed-groups
• nginx.ingress.kubernetes.io/auth-url

Limitations

• Does not support exposing services using Service resources with NodePort, LoadBalancer, or ExternalName types
• Request body size limit is 10MB

1. If your browser contains valid authentication cache, the login process will be skipped. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Exposing Workloads as Web APIs

The API Identity-Aware Proxy (API IAP) feature in PFCP allows you to expose your workloads as web APIs available over the internet. The exposed web APIs will automatically implement authentication for access, requiring a Kubernetes ServiceAccount token for authorization.

This section explains how to expose your web APIs over the internet using API IAP.

Note

For deploying your workloads as web applications accessible via browsers, please refer to Exposing Workloads as Web Applications.

Supported Protocols

The following protocols are supported.

• HTTP/2
• HTTP/1.1

Note

• HTTP/2 is terminated by API IAP and forwarded to the Service as HTTP/1.1
• gRPC is not supported

Exposing Web APIs Using Ingress Resources

1. Prepare the workload and Service resources you wish to expose. For this example, assume you can access the workload by connecting to port 80 of the example-svc Service.

2. Create an Ingress manifest using the following template:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: example-ingress
   spec:
     rules:
     - # Specifies the domain to assign to the Ingress.
       # Note: Unlike web application deployments, you must specify a subdomain of `api.iap.pfcomputing.com`.
       host: example.<organization-name>.<cluster-name>.api.iap.pfcomputing.com
       http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             # Specifies the name of the Service and its port to expose.
             service:
               name: example-svc
               port:
                 number: 80
   

   Warning

   Subdomain Restrictions

   For web API Ingresses, only the domain *.<organization-name>.<cluster-name>.api.iap.pfcomputing.com can be used.

   For example, if your organization name is foo and cluster name is sr1-01, the valid subdomain would be *.foo.sr1-01.api.iap.pfcomputing.com.

3. Create a Kubernetes ServiceAccount for accessing the Ingress using the following template:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: example-sa
     # Specify the same Namespace as the Ingress.
     namespace: org-foo
     labels:
       # Specifies the name of the Ingress resource to allow access.
       ingress.preferred.jp/allowed-ingress: example-ingress
   

   Warning

   Default ServiceAccount

   The Kubernetes-automated default ServiceAccount cannot be used for accessing Ingress resources.

4. Generate a Kubernetes ServiceAccount token using kubectl. The Audience field must be set to the domain configured for the Ingress resource, with https:// prepended. You may adjust the token’s expiration period as needed.

   $ kubectl create token example-sa --duration 12h --audience https://example.<organization-name>.<cluster-name>.api.iap.pfcomputing.com
   eyJ...
   

   Note

   Token Expiration

   Tokens with infinite validity cannot be issued. Instead, specify a sufficiently long duration.

5. Access the configured domain by including the generated token in the Authorization HTTP header. Verify that you can successfully access the exposed service.

   $ curl --dump-header - -H "Authorization: Bearer eyJ..." https://example.<organization-name>.<cluster-name>.api.iap.pfcomputing.com
   

Managing Tokens

To deactivate a previously issued Kubernetes ServiceAccount token, simply delete the corresponding ServiceAccount. For granular token deactivation per token, consider using Bound ServiceAccount Tokens.

Removing the ingress.preferred.jp/allowed-ingress label from a Kubernetes ServiceAccount can temporarily block access without deactivating the token. To re-enable access, simply reapply the label.

You can check the list of Kubernetes ServiceAccounts with Ingress access permissions using the kubectl command:

$ kubectl get serviceaccount --selector ingress.preferred.jp/allowed-ingress
NAME         SECRETS   AGE
example-sa   0         4s

Limitations

• Does not support exposing services using Service resource types NodePort, LoadBalancer, or ExternalName
• If multiple Ingress resources are configured with the same domain matching the token’s Audience, authentication will fail with a 401 Unauthorized status code
• Connections will be terminated after 24 hours of inactivity
• Request body size is limited to 10MB

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Automatic Horizontal Scaling of Workloads and Jobs Based on External Events

PFCP provides functionality for automatically scaling workloads and jobs horizontally using external event-driven triggers, utilizing KEDA (Kubernetes Event Driven Autoscaler).

Available KEDA Resources

The following three KEDA resources are supported for use with PFCP:

ScaledObject

The ScaledObject resource defines both the target workload to be scaled (e.g., Deployment, StatefulSet) and the triggering mechanism that controls the scaling process. For detailed information, please refer to the official ScaledObject documentation.

ScaledJob

The ScaledJob resource defines the target job to be scaled and the triggering mechanism that controls its scaling. For detailed information, please refer to the official ScaledJob documentation.

TriggerAuthentication

The TriggerAuthentication resource manages authentication credentials for external systems. For detailed information, please refer to the official TriggerAuthentication documentation.

Scaling Using PFCP Prometheus Metrics

PFCP offers Prometheus-based metrics monitoring functionality. Below we explain how to use this feature to automatically scale the number of Pod replicas for a Deployment.

The following is an example KEDA configuration for scaling an application based on Prometheus metrics. It creates a ScaledObject resource targeting a Deployment named myapp. Since the Prometheus query always returns 100 with a threshold of 50, the Pod replica count will be scaled to 2.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: app-scaler
spec:
  scaleTargetRef:
    name: myapp # Specifies the name of the Deployment to be scaled
  maxReplicaCount: 3 # Specifies the maximum number of replicas
  minReplicaCount: 0 # Specifies the minimum number of replicas
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring-org-<org-name>.svc.cluster.local.:9090 # org-name: Organization name
        query: "vector(100)" # Specifies the metric to monitor (here, a query that always returns 100)
        threshold: "50" # Specifies the scaling condition (here, when the metric exceeds 50)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - image: stefanprodan/podinfo
        name: podinfo
        ports:
        - containerPort: 9898

For instructions on collecting Pod metrics with Prometheus, please refer to Metrics Monitoring and Alerting.

Scaling Using AWS Managed Prometheus Metrics

This section provides an example of automatically scaling the number of Pod replicas for a Deployment by utilizing AWS’s Managed Prometheus (AMP) metrics as an external resource.

AWS IAM Configuration

To use AMP metrics, KEDA must be configured to query AMP. The following steps outline the required configuration:

1. Create an OpenID Connect (OIDC) identity provider in IAM by following the instructions in Configuring Identity Federation with Public Clouds.
2. Note the ARN of your AMP workspace in use.

   For AMP setup instructions, please refer to the AWS documentation.

3. Create a policy that allows KEDA to query AMP metrics. Replace <aws_region> and <aws_account_id> with your actual AMP workspace ARN.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "aps:QueryMetrics"
               ],
               "Resource": [
                   "arn:aws:aps:<aws_region>:<aws_account_id>:workspace/<workspace_id>"
               ]
           }
       ]
   }
   

4. Create an IAM role to attach this policy to and associate it with the KEDA system components. Set the Principal to the OIDC provider created in step 1. The Namespace for the KEDA Operator in the Condition varies by organization. Below is an example:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Principal": {
           "Federated": "arn:aws:iam::<aws_account_id>:oidc-provider/token.<pfcp_cluster>.kubernetes.pfcomputing.com"
         },
         "Condition": {
           "StringEquals": {
             "token.<pfcp_cluster>.kubernetes.pfcomputing.com:aud": [
               "sts.amazonaws.com"
             ],
             "token.<pfcp_cluster>.kubernetes.pfcomputing.com:sub": [
               "system:serviceaccount:keda-org-<pfcp_org_name>:keda-operator"
             ]
           }
         }
       }
     ]
   }
   

5. Note the ARN of the created role. This ARN will be used later in KEDA configuration.

Creating KEDA Resources

The following is an example KEDA configuration for scaling an application based on AMP metrics.

By specifying the ARN of the IAM role created earlier in .spec.podIdentity.roleArn for TriggerAuthentication, KEDA will use that role to query AMP metrics.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: amp-query-role
spec:
  podIdentity:
    provider: aws
    roleArn: <roleArn> # Specifies the ARN of the IAM role created earlier
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: app-scaler
spec:
  scaleTargetRef:
    name: myapp
  maxReplicaCount: 10
  minReplicaCount: 0
  triggers:
    - type: prometheus
      authenticationRef:
        name: amp-query-role
      metadata:
        awsRegion: <aws_region> # Specifies the AMP region
        serverAddress: https://aps-workspaces.<aws_region>.amazonaws.com/workspaces/<workspace_id> # Specifies the AMP endpoint
        query: "sum(rate(http_requests_total[1m]))" # Specifies the metric to monitor (here, HTTP requests per minute)
        threshold: "50" # Specifies the scaling condition (here, when the request count exceeds 50)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - image: stefanprodan/podinfo
        name: podinfo
        ports:
        - containerPort: 9898

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Managing Sensitive Data (Secrets)

Kubernetes provides native support for Secrets through its core functionality.

While Secret resources are designed to manage sensitive data, they are typically handled in API calls and manifests as base64-encoded data. This data is not encrypted and can be easily decoded to plaintext, which presents security risks. Typically, Kubernetes resource manifests are managed in git repositories using GitOps or Infrastructure as Code (IaC) practices in YAML format. However, since Secret data can be easily decoded to plaintext, storing it directly in a git repository is inappropriate.

PFCP addresses this limitation by offering SealedSecret, a mechanism for encrypting sensitive data so that it can be managed alongside regular Kubernetes manifests using GitOps methods.

Overview

A SealedSecret is a resource containing encrypted sensitive data that can be stored in a git repository.

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: mynamespace
spec:
  encryptedData:
    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq..... # Encoded encrypted sensitive data

When this SealedSecret resource is applied to the cluster, it is decrypted using cluster-internal keys and transformed into a standard Secret resource.

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: mynamespace
data:
  foo: YmFy # Base64-encoded "bar"

Creating a SealedSecret

To create a SealedSecret, you can use the kubeseal command. There are multiple installation methods for kubeseal; please refer to the sealed-secrets documentation. We recommend using releases maintained by Bitnami.

First, create a regular Kubernetes Secret and save it to a file. You can use the kubectl command for this:

echo -n secret-data | kubectl create secret generic mysecret \
    --dry-run=client \
    --from-file=secret-name=/dev/stdin \
    -o yaml >secret.yaml

This will generate a file with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
data:
  secret-name: c2VjcmV0LWRhdGE=

Next, use the kubeseal command to create a SealedSecret. kubeseal retrieves the public key (certificate) from the sealed-secrets controller running in your cluster, encrypts the Secret using this public key, and outputs the SealedSecret.

kubeseal \
    --controller-namespace sealed-secrets-org-<organization-name> \
    -f secret.yaml \
    -o yaml > sealedsecret.yaml

This will generate a file with the following content:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
spec:
  encryptedData:
    secret-name: "AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq....."

You can now store this sealedsecret.yaml file in your git repository.

Note

The secret.yaml file contains unencrypted sensitive data and should never be stored in a git repository.

git add sealedsecret.yaml
git commit -m "add new secret"

When applying the SealedSecret to your cluster, it will be automatically decrypted by the cluster and can be used just like a regular Kubernetes Secret. For detailed information, please refer to the Kubernetes documentation.

Managing Public and Private Keys

The key pair (public and private keys) used for encrypting and decrypting sensitive data is organization-specific and is primarily managed by PFCP. Users do not need to manage these keys themselves. The public and private keys are automatically rotated at regular intervals. Rest assured that the private key exists only within the cluster.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Configuring Identity Federation with Public Cloud Services

The keys used for accessing public cloud resources are strong authentication credentials, and unauthorized removal poses significant security risks. To enable access to public cloud resources without transferring these keys, PFCP supports identity federation with public cloud providers. Through identity federation, permissions for the public cloud are granted to Kubernetes ServiceAccounts, allowing secure access.

This section provides instructions for establishing identity federation connections with two major public cloud providers: Amazon Web Services (AWS) and Google Cloud.

Configuring with AWS

AWS-Side Configuration

1. Refer to the AWS documentation Creating an OpenID Connect (OIDC) Identity Provider in IAM - AWS Identity and Access Management to create an OIDC provider within your target AWS account. ¹

   • For the provider URL, refer to the Region and Compute Node documentation.
   • Allowed audience: sts.amazonaws.com

2. Create an IAM role to be used for accessing AWS. This IAM role will be linked to the Kubernetes ServiceAccount through identity federation.

3. Configure a trust policy specifying the Kubernetes ServiceAccount that should assume this role to enable federation-based linking (see documentation):

   {
       "Version": "2012-10-17",
       "Statement": [{
           "Sid": "",
           "Effect": "Allow",
           "Principal": {
               // Specify the ARN of the OIDC provider created in the previous step.
               "Federated": "arn:aws:iam::XXXXXXXXXXX:oidc-provider/token.<pfcp_cluster>.kubernetes.pfcomputing.com"
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
               "StringEquals": {
                   // Specify the name of the Kubernetes ServiceAccount to assume this role.
                   "token.<pfcp_cluster>.kubernetes.pfcomputing.com:sub": "system:serviceaccount:<namespace>:<serviceaccount>"
               }
           }
       }]
   }
   

   Warning

   Ensure you properly configure the Condition section. Failure to do so will grant permissions to all ServiceAccounts, including those from other organizations.

4. By default, AWS enables audit logging for identity federation through Cloud Trail and retains logs for 90 days. To extend retention beyond 90 days, refer to Creating a Trail for Your AWS Account - AWS CloudTrail and modify the settings accordingly.

Kubernetes Cluster-Side Configuration

1. Specify the AWS IAM role to be assigned to the ServiceAccount in the ServiceAccount annotations:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: foo
     annotations:
       aws.id-federation.preferred.jp/role-arn: "arn:aws:iam::{aws_account_id}:role/{role}"
   

2. When creating Pods, specify this ServiceAccount in the .spec.serviceAccountName field. Using AWS SDKs within the Pod will grant access to AWS via the AWS IAM Role associated with the ServiceAccount. The session token expires after 1 hour and will be automatically refreshed as long as the AWS SDK remains in use.

3. Verify the configuration by checking:

   • The IAM role currently used by the Pod:

     $ kubectl get po <pod> -o yaml | grep AWS_ROLE_ARN:
         AWS_ROLE_ARN:                 arn:aws:iam::861856390547:role/id-federation-test-sr1-01
     

   • The presence of a web identity token file mounted in the Pod:

     $ kubectl get po <pod> -o yaml | grep AWS_WEB_IDENTITY_TOKEN_FILE:
         AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/sts.amazonaws.com/serviceaccount/token
     

Configuring with Google Cloud

PFCP deploys gcp-workload-identity-federation-webhook, which can be used to configure identity federation with Google Cloud.

Google Cloud-Side Configuration

Refer to Configuring Workload Identity Integration with Kubernetes | IAM Documentation | Google Cloud and perform the following setup:

1. Create a Workload Identity pool and provider within the Google Cloud project you wish to access.

   • For the provider URL, refer to the Region and Compute Node documentation. ¹

2. Create a Google IAM service account to be used for accessing Google Cloud and grant it necessary permissions for the target resources. This service account will be linked to the Kubernetes ServiceAccount through identity federation.

3. To enable federation-based linking, assign the Workload Identity user role (roles/iam.workloadIdentityUser) to the above Google IAM service account. When specifying members, limit permission to only the Kubernetes ServiceAccount by including the following:

   principal://iam.googleapis.com/projects/<Google Cloud project ID> \
   /locations/global/workloadIdentityPools/<ID of created Workload Identity pool> \
   /subject/<cluster's provider URL>::system:serviceaccount:<Kubernetes ServiceAccount Namespace>:<Kubernetes ServiceAccount Name>
   

Kubernetes Cluster-Side Configuration

1. In the ServiceAccount annotations, configure the following values: the Workload Identity provider, the IAM service account to be assumed, and the expected audience value required by the Workload Identity provider.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     annotations:
       googlecloud.id-federation.preferred.jp/workload-identity-provider: |-
         projects/<Google Cloud project ID>/locations/global/workloadIdentityPools/<ID of created Workload Identity pool>/providers/<ID of created Workload Identity pool provider>
       googlecloud.id-federation.preferred.jp/service-account-email: |-
         <Google IAM service account name>@<Google Cloud project name>.iam.gserviceaccount.com
       googlecloud.id-federation.preferred.jp/audience: <Expected audience value for Workload Identity provider>
   

2. When creating Pods, specify this ServiceAccount in the .spec.serviceAccountName field. Using Google Cloud SDKs within the Pod will automatically access Google Cloud using the Google IAM service account linked to the specified ServiceAccount. The credentials default to 24-hour validity and will be automatically refreshed as long as the Google Cloud SDK remains in use.

For more detailed information, please refer to the README for the gcp-workload-identity-federation-webhook.

1. If you are using multiple PFCP clusters, separate configuration is required for each cluster. ↩ ↩2

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Running GitHub Actions Jobs

PFCP enables the execution of GitHub Actions jobs on clusters. This allows users to build comprehensive CI/CD pipelines that leverage cluster resources.

For executing GitHub Actions jobs, PFCP provides the Actions Runner Controller (ARC). Using ARC, you can deploy self-hosted runners directly on your cluster to run GitHub Actions jobs.

Below we outline the process for executing GitHub Actions jobs with ARC.

Configuring Self-Hosted Runners

First, follow the steps below to set up self-hosted runners on your cluster. The number of runners will automatically scale based on the number of queued jobs.

1. Create a GitHub App and install it on the target repository. Also, create a Kubernetes Secret to store the GitHub App’s authentication credentials.

   • For detailed instructions, refer to Authenticating ARC to the GitHub API - GitHub Documentation.

2. Create a AutoscalingRunnerSet resource in your cluster. Below is an example configuration:

   apiVersion: actions.github.com/v1alpha1
   kind: AutoscalingRunnerSet
   metadata:
     name: NAME
   spec:
     githubConfigUrl: https://github.com/ORG/REPO # The repository where runners will be registered
     githubConfigSecret: SECRET_NAME # The name of the Secret created in step 1
     runnerScaleSetName: RUNNER_NAME # The name specified in the "runs-on" field for GitHub Actions jobs
     # Configuration for automatic runner scaling
     minRunners: 0
     maxRunners: 20
     # Template for runner pods
     template:
       spec:
         containers:
         - name: runner
           image: ghcr.io/actions/actions-runner
           command:
           - /home/runner/run.sh
           # Adjust resource limits according to the jobs you intend to run
           resources:
             requests:
               cpu: 100m
             limits:
               memory: 256Mi
   

   Note

   The app.kubernetes.io/version label and .spec.listenerTemplate field of the AutoscalingRunnerSet resource are automatically configured. Any existing values will be overwritten.

Using with Your GitHub Actions Job

To utilize the self-hosted runners you’ve created in your GitHub Actions job, follow these steps:

1. In the job’s runs-on property, specify the name set in the runnerScaleSetName of the AutoscalingRunnerSet.

   jobs:
     build:
       runs-on: RUNNER_NAME
       steps:
       - ...
   

References

For more detailed information about Actions Runner Controller, please refer to the official documentation.

• Actions Runner Controller - GitHub Documentation
• Deploying runner scale sets with Actions Runner Controller - GitHub Documentation

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Running Relational Databases

PFCP provides MOCO to simplify the deployment and management of MySQL workloads.

Note

Workloads like MySQL created by MOCO are deployed on Compute Nodes alongside other organizational workloads.

Available MOCO Resources

PFCP supports the following two MOCO resources:

MySQLCluster

MySQLCluster is a resource for defining MySQL cluster configurations. Creating this resource deploys a MySQL cluster managed by MOCO. For detailed information, please refer to the official MOCO documentation: Creating clusters.

BackupPolicy

BackupPolicy is a resource for managing MySQL backups. Creating this resource enables configuration of regular backups for MySQL clusters. MOCO specifies backup destinations using bucketConfig. For backendType, you can specify s3, gcs, or azure. For PFCP deployments, you must first prepare object storage accessible by the cluster along with authentication settings for accessing that object storage. For detailed information, please refer to the official MOCO documentation: Backup and Restore.

Deploying a MySQL Instance

Below is an example MySQLCluster resource for deploying a MySQL instance. Be sure to specify the StorageClass name for storageClassName as standard-rwo-<organization-name>. Since MySQL’s data directory is written from a single Pod, we recommend using block storage with ReadWriteOncePod instead of RWX file storage.

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  name: moco-test-instance
spec:
  replicas: 1  # Change replicas to configure replication
  podTemplate:
    spec:
      containers:
      - name: mysqld
        image: ghcr.io/cybozu-go/moco/mysql:8.4.8
        resources:
          limits:
            cpu: "2"
            memory: "8Gi"
  volumeClaimTemplates:
  - metadata:
      name: mysql-data
    spec:
      accessModes:
      - ReadWriteOncePod
      storageClassName: standard-rwo-<組織名>
      resources:
        requests:
          storage: 4Gi

Configuring Backups

A MySQLCluster referencing a BackupPolicy will automatically create a CronJob moco-backup-<MySQLCluster-name> for backups.

The backup job will run under the ServiceAccount specified by spec.jobConfig.serviceAccountName. Authentication credentials for accessing object storage should be provided either through the ServiceAccount or via env/envFrom. If you’re using identity federation with public clouds with PFCP, please also refer to Configuring Identity Federation with Public Clouds.

workVolume serves as a temporary workspace for storing dump files and compressed data. It is not intended for storing actual backups. Using Pod local storage options like emptyDir or ephemeral may not provide sufficient temporary space for backup operations. For larger backup targets, we recommend configuring a separate work volume PVC distinct from the MySQL data PVC and referencing it via workVolume.

Below is an example BackupPolicy configuration that stores daily backups in Google Cloud Storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: moco-backup-work
spec:
  accessModes:
  - ReadWriteOncePod
  storageClassName: standard-rwo-<組織名>
  resources:
    requests:
      storage: 20Gi
---
apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  name: daily-backup
spec:
  schedule: "@daily"
  jobConfig:
    serviceAccountName: db-backup-sa
    bucketConfig:
      bucketName: example-moco-backup
      endpointURL: https://storage.googleapis.com
      backendType: gcs
    workVolume:
      persistentVolumeClaim:
        claimName: moco-backup-work

moco-backup-work is a PVC for temporary workspace used by the backup Job. Ensure this is configured separately from the MySQL data storage area. On the MySQLCluster side, specify the BackupPolicy name in spec.backupPolicyName. Since MOCO does not automatically delete old backups, you should also configure lifecycle settings for the backup destination bucket as needed.

Connecting to the MySQL Instance

MOCO creates two Services for each MySQLCluster: a write-access moco-<MySQLCluster-name>-primary Service and a read-only moco-<MySQLCluster-name>-replica Service. For example, if you create a moco-test-instance in the org-<organization-name> Namespace, you can use moco-moco-test-instance-primary.org-<organization-name>.svc and moco-moco-test-instance-replica.org-<organization-name>.svc.

Additionally, MOCO provides standard MySQL users: moco-readonly, moco-writable, and moco-admin. For obtaining credentials or interactive connections, the kubectl-moco kubectl plugin is particularly useful.

$ kubectl moco -n org-<組織名> credential -u moco-admin moco-test-instance --format mycnf
$ kubectl moco -n org-<組織名> mysql -it moco-test-instance
$ kubectl moco -n org-<組織名> mysql -u moco-writable moco-test-instance -- -e "CREATE DATABASE app"

When connecting from application Pods, use moco-<MySQLCluster-name>-primary for write operations and moco-<MySQLCluster-name>-replica for read operations. For detailed instructions, please refer to the official MOCO documentation: Using the Cluster.

Note

Best practices for operating MySQL

• Always use MySQL versions supported by MOCO. You can perform MySQL upgrades using the MOCO upgrade features.
• For enhanced availability, configure replicas to 2 or higher to enable replication.
• For persistent storage requirements, use the BackupPolicy to create backups.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Metrics Monitoring and Alerting

PFCP provides a managed service for metrics monitoring and alerting using Grafana¹ and Prometheus². Additionally, we have implemented the Prometheus Operator, enabling declarative management of monitoring targets and alert rules through Kubernetes custom resources.

Accessing Monitoring Services

You can access all monitoring service features through the links provided on the portal homepage.

Grafana Dashboards

We provide standard Grafana dashboards for visualizing the status of Kubernetes workloads. The standard dashboards include:

• kube-prometheus > Kubernetes / Compute Resources / Namespace (Workloads)
  • Resource usage by namespace
• kube-prometheus > Kubernetes / Compute Resources / Pod
  • Resource usage by pod

You can also create custom dashboards directly from the Grafana Web UI. We recommend creating dedicated folders for adding new dashboards and organizing them within those folders.

Metrics Monitoring

Scraping Pod Metrics

You can use the ServiceMonitor and PodMonitor custom resources from the Prometheus Operator to scrape metrics from pods. For visualizing the collected metrics, you can use Grafana dashboards.

For details on using ServiceMonitor and PodMonitor, please refer to the official Prometheus Operator documentation.

• ServiceMonitor
• PodMonitor

Adding Alert Rules

We provide standard alert rules to check the health status of Kubernetes workloads. The default alert rules can be viewed in the Alerts tab of the Prometheus Web UI.

Additionally, you can add custom alert rules using the PrometheusRule custom resource from the Prometheus Operator. For details on using PrometheusRule, please refer to the following documentation:

• PrometheusRule

Adding Alert Notification Destinations

You can use the AlertmanagerConfig custom resource from the Prometheus Operator to send alerts to any desired services or tools. For details on using AlertmanagerConfig, please refer to the following documentation:

• AlertmanagerConfig

Sample Manifest File

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmanager-config
spec:
  inhibitRules:
  - equal:
    - namespace
    - alertname
    sourceMatch:
    - name: severity
      value: critical
    targetMatch:
    - matchType: =~
      name: severity
      value: warning|info
  - equal:
    - namespace
    - alertname
    sourceMatch:
    - name: severity
      value: warning
    targetMatch:
    - name: severity
      value: info
  - equal:
    - namespace
    sourceMatch:
    - name: alertname
      value: InfoInhibitor
    targetMatch:
    - name: severity
      value: info
  route:
    groupBy:
    - alertname
    groupInterval: 5m
    groupWait: 30s
    receiver: slack
    repeatInterval: 12h
    routes:
    - matchers:
      - name: alertname
        value: InfoInhibitor
      receiver: "null"
    - matchers:
      - name: alertname
        value: Watchdog
      receiver: "null"
  receivers:
  - name: "null"
  - name: slack
    slackConfigs:
    - apiURL:
        name: alertmanager-cred
    key: slack-url
      sendResolved: true

Prometheus Server Specifications

The specifications of the provided Prometheus server are as follows:

• Metrics retention period: 15 days
• Metrics retention size: 95GiB

Warning

We do not support modifications to the metrics retention period or retention size.

Limitations

• Grafana and Prometheus do not provide organizational-level permission management functionality. Regular users can view all metrics and dashboards.
• For security reasons, the spec.endpoints[].bearerTokenFile field cannot be used in the ServiceMonitor custom resource. Instead, the spec.endpoints[].authorization field can be used to specify tokens.

1. Grafana is an open-source dashboard tool for visualizing time-series data from various data sources. https://grafana.com/docs/grafana/latest ↩

2. Prometheus is an open-source monitoring and alerting system specialized for collecting and querying time-series data. https://prometheus.io/docs/ ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Viewing Kubernetes Resources in the Dashboard

PFCP provides Headlamp as the web-based user interface for Kubernetes clusters. With Headlamp, you can inspect Kubernetes resources accessible under your user permissions directly from the browser. Additionally, depending on the resource, you may be able to perform operations such as creation, update, and deletion.

Access Method

1. Log in to the PFCP Portal.
2. On the main page, select the target cluster and click the Headlamp link.

Note

The Headlamp URL follows the format https://<organization-name>.<cluster-name>.headlamp.pfcomputing.com. Typically, you should access it via the portal link.

Primary Usage Features

Headlamp allows you to view the following information:

• Status of resources including Pods, Deployments, Jobs, and Services
• Information about ReservedNodes allocated to your organization
• Detailed event and manifest information in the resource detail view

Furthermore, the Pod detail screen offers Prometheus metrics display functionality.

Limitations

• The resources and operations you can view/perform depend on your Kubernetes permissions assigned for the user.

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Applying Security Policies to Pods

Kubernetes defines Pod Security Standards (hereafter PSS) as recommended configurations for Pod security to reduce security risks. PFCP applies Pod security profiles compliant with these standards.

PSS provides three policy options: Restricted, Baseline, and Privileged. PFCP defaults to the Baseline policy, which maintains relatively relaxed restrictions on container workloads while providing protection against known privilege escalation vulnerabilities.

For enhanced Pod security, the Restricted policy is also available. By adding the policy.preferred.jp/pod-security: restricted label to a Pod, you enforce the Restricted policy and prevent the creation of Pods that violate its security requirements.

Note

The Privileged policy, which permits privilege escalation, is unavailable.

For detailed information about each PSS policy, please refer to the official Kubernetes documentation.

Profile Usage for the MN-Core Series

When creating Pods that request resources from the MN-Core series, automatic assignment of the SYS_NICE capability is performed to accelerate computation using MN-Core series hardware¹. This enhancement applies exclusively to Pods requesting MN-Core series resources. For Pods that do not request MN-Core series resources, compliance with the PSS Baseline policy prevents the assignment of SYS_NICE capability.

1. This is not a standard Kubernetes feature but is a unique extension provided by PFCP. ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Specifications: Data Center and Compute Nodes

Data Center

SR1

• Source IP address for internet communications: 202.214.130.143/32

Cluster

• SR1-01
  • Provider (Issuer) URL for ID Federation: https://token.sr1-01.kubernetes.pfcomputing.com

IK1

• Source IP address for internet communications: 153.120.101.128/28
  • Note: This address may change without prior notice.

Cluster

• IK1-01
  • Provider (Issuer) URL for ID Federation: https://token.ik1-01.kubernetes.pfcomputing.com

Compute Nodes

MN-Server 2 V1 (Node Name: isXXzXcqXX)

For more detailed specifications, please refer to the MN-Server 2 V1 entry in the MN-Core 2 Hardware Catalog.

1. Intel Xeon Platinum 8480+ Product Specifications ↩

2. NVIDIA ConnectX-6 Dx EN Adapter Card User Manual ↩

Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Specifications: Software Versions

Cluster Components and Add-ons

Monitoring Services