Warning

This page was translated from the original Japanese version by PLaMo Translate. The Japanese version is authoritative; the English translation may contain inaccuracies.

Metrics Monitoring and Alerting

PFCP provides a managed service for metrics monitoring and alerting using Grafana¹ and Prometheus². Additionally, we have implemented the Prometheus Operator, enabling declarative management of monitoring targets and alert rules through Kubernetes custom resources.

Accessing Monitoring Services

You can access all monitoring service features through the links provided on the portal homepage.

Grafana Dashboards

We provide standard Grafana dashboards for visualizing the status of Kubernetes workloads. The standard dashboards include:

• kube-prometheus > Kubernetes / Compute Resources / Namespace (Workloads)
  • Resource usage by namespace
• kube-prometheus > Kubernetes / Compute Resources / Pod
  • Resource usage by pod

You can also create custom dashboards directly from the Grafana Web UI. We recommend creating dedicated folders for adding new dashboards and organizing them within those folders.

Metrics Monitoring

Scraping Pod Metrics

You can use the ServiceMonitor and PodMonitor custom resources from the Prometheus Operator to scrape metrics from pods. For visualizing the collected metrics, you can use Grafana dashboards.

For details on using ServiceMonitor and PodMonitor, please refer to the official Prometheus Operator documentation.

• ServiceMonitor
• PodMonitor

Adding Alert Rules

We provide standard alert rules to check the health status of Kubernetes workloads. The default alert rules can be viewed in the Alerts tab of the Prometheus Web UI.

Additionally, you can add custom alert rules using the PrometheusRule custom resource from the Prometheus Operator. For details on using PrometheusRule, please refer to the following documentation:

• PrometheusRule

Adding Alert Notification Destinations

You can use the AlertmanagerConfig custom resource from the Prometheus Operator to send alerts to any desired services or tools. For details on using AlertmanagerConfig, please refer to the following documentation:

• AlertmanagerConfig

Sample Manifest File

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmanager-config
spec:
  inhibitRules:
  - equal:
    - namespace
    - alertname
    sourceMatch:
    - name: severity
      value: critical
    targetMatch:
    - matchType: =~
      name: severity
      value: warning|info
  - equal:
    - namespace
    - alertname
    sourceMatch:
    - name: severity
      value: warning
    targetMatch:
    - name: severity
      value: info
  - equal:
    - namespace
    sourceMatch:
    - name: alertname
      value: InfoInhibitor
    targetMatch:
    - name: severity
      value: info
  route:
    groupBy:
    - alertname
    groupInterval: 5m
    groupWait: 30s
    receiver: slack
    repeatInterval: 12h
    routes:
    - matchers:
      - name: alertname
        value: InfoInhibitor
      receiver: "null"
    - matchers:
      - name: alertname
        value: Watchdog
      receiver: "null"
  receivers:
  - name: "null"
  - name: slack
    slackConfigs:
    - apiURL:
        name: alertmanager-cred
    key: slack-url
      sendResolved: true

Prometheus Server Specifications

The specifications of the provided Prometheus server are as follows:

• Metrics retention period: 15 days
• Metrics retention size: 95GiB

Warning

We do not support modifications to the metrics retention period or retention size.

Limitations

• Grafana and Prometheus do not provide organizational-level permission management functionality. Regular users can view all metrics and dashboards.
• For security reasons, the spec.endpoints[].bearerTokenFile field cannot be used in the ServiceMonitor custom resource. Instead, the spec.endpoints[].authorization field can be used to specify tokens.

1. Grafana is an open-source dashboard tool for visualizing time-series data from various data sources. https://grafana.com/docs/grafana/latest ↩

2. Prometheus is an open-source monitoring and alerting system specialized for collecting and querying time-series data. https://prometheus.io/docs/ ↩