Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

PFCP Tutorial: Importing Data from Cloud Storage to Your Cluster

This page provides step-by-step instructions for transferring data from cloud storage to PFCP.

As an example, we will demonstrate how to copy data from an AWS S3 bucket managed by the user to a Persistent Volume within PFCP.

Preliminary Setup

Please complete the following steps from the “Connect to Your Cluster” section in the left column:

You are logged in to the PFCP portal
Your cluster connection has been properly configured

Additionally, you will need an AWS account and the target S3 bucket where you want to synchronize your data¹.

Configuring Public Cloud Identity Federation and Copying Data

AWS Configuration

These instructions assume you are using the sr1-01 region. If you are in a different region, please adjust the domain names accordingly.

Create an OIDC provider within the AWS account you wish to access².
- For the provider (issuer) URL, refer to the Regions and Compute Nodes documentation.
- Allowed audience: sts.amazonaws.com
Create an IAM role that will be used to access AWS. This IAM role will be linked to a Kubernetes ServiceAccount through identity federation. Here, we will create an IAM role named data-transfer-sr1-01.

Configure a trust policy that specifies the Kubernetes ServiceAccount to which this role should be bound for identity federation³. You can either use an existing ServiceAccount or create one later. For this example, we will create a ServiceAccount named data-transfer-sa.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            // Specify the ARN of the oidc provider created in the previous step.
            "Federated": "arn:aws:iam::{aws_account_id}:oidc-provider/token.sr1-01.kubernetes.pfcomputing.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
            "StringEquals": {
                // Specify the name of the Kubernetes ServiceAccount to which this role should be bound.
                "token.sr1-01.kubernetes.pfcomputing.com:sub": "system:serviceaccount:<namespace>:data-transfer-sa"
            }
        }
    }]
}

Configure a policy on the S3 bucket containing the data you wish to transfer to allow access from a specific IAM role⁴.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{aws_account_id}:role/data-transfer-sr1-01"
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
      ],
      "Resource": [
        "arn:aws:s3:::example-bucket",
        "arn:aws:s3:::example-bucket/*"
      ]
    }
  ]
}

Kubernetes Cluster Configuration

Create a ServiceAccount. Specify the AWS IAM role data-transfer-sr1-01 with access to your S3 bucket in the annotations of the ServiceAccount you will use for data copying.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: data-transfer-sa
  annotations:
    aws.id-federation.preferred.jp/role-arn: "arn:aws:iam::{aws_account_id}:role/data-transfer-sr1-01"

Verify that identity federation is functioning correctly. When creating a Pod, specify this ServiceAccount in the spec.serviceAccountName field. The Pod will automatically be configured with an AWS session token valid for one hour. When the token expires, the AWS SDK will automatically refresh it. Run the aws sts get-caller-identity command to verify that the session token can successfully assume the AWS IAM Role bound to the ServiceAccount.
```
$ kubectl run --rm --overrides='{"spec":{"serviceAccountName": "data-transfer-sa"}}' id-federation-check --image=amazon/aws-cli -- sts get-caller-identity
  {
      "UserId": "***********************:botocore-session-1751604047",
      "Account": "{aws_account_id}",
      "Arn": "arn:aws:sts::{aws_account_id}:assumed-role/data-transfer-sr1-01/botocore-session-1751604047"
  }
```

Create a Persistent Volume Claim (PVC) to store your data⁵.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-transfer-pvc
spec:
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard-rwx-<organization-name>

Create a Job to copy data using the AWS CLI’s s3 sync command. Mount the PVC created above and specify data-transfer-sa in the template.spec.serviceAccountName field, ensuring it has enabled identity federation with AWS. Replace <bucket-name> and <object-name> with appropriate values.

apiVersion: batch/v1
kind: Job
metadata:
  name: data-transfer-job
spec:
  template:
    spec:
      serviceAccountName: data-transfer-sa # Specifies the ServiceAccount configured for identity federation
      containers:
      - name: transfer
        image: amazon/aws-cli
        command: ["aws"]
        args: ["s3", "sync", "s3://<bucket-name>/<object-name>", "/mnt/data"]
        volumeMounts:
        - mountPath: "/mnt/data"
          name: my-volume
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: data-transfer-pvc
      restartPolicy: OnFailure

Run kubectl logs job/data-transger-job to verify that the command completed successfully.

$ kubectl logs job/data-transger-job
    download: s3://<bucket-name>/file1.txt to /mnt/data/file1.txt
    ...
    Completed 10 of 10 file(s), 100% done.

Cleanup

Delete all resources created during this tutorial.

kubectl delete job/data-transfer-job
kubectl delete pvc/data-transfer-pvc
kubectl delete serviceaccount/data-transfer-sa

If necessary, also delete the corresponding AWS resources.