Claire's Blog

使用套件

1. Prometheus
2. Prometheus Operator
3. K8S
4. Rancher

步驟一、設定自訂義指標

1. 設定Exporter，這邊有許多官方提供的函式庫
https://prometheus.io/docs/instrumenting/clientlibs/
2. 設定該export的service名為my-export

apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-export
  name: my-export
  namespace: default
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: my-export
    port: 7979
    protocol: TCP
    targetPort: 7979
  selector:
    prometheus-customized-metrix: my-export
  type: ClusterIP

設定Service Monitors

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: my-export
  name: my-export
  namespace: default
spec:
  endpoints:
  - interval: 30s
    params:
      module:
      - default
      target:
      - http://127.0.0.1:1985/api/v1/streams/
    path: /probe
    port: my-export
  jobLabel: jobLabel
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: my-export

步驟二、設定自訂義規則

增加Prometheus Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app: rancher-monitoring
    app.kubernetes.io/instance: rancher-monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: rancher-monitoring
    app.kubernetes.io/version: 16.6.1_up16.6.0
    chart: rancher-monitoring-16.6.1_up16.6.0
    heritage: Helm
    release: rancher-monitoring
  name: my-data
  namespace: default
spec:
  groups:
  - name: my-data
    rules:
    - expr: sum(mydata{container="my-container", name=~".+",namespace=~"default"})
        by (pod)
      labels:
        namespace: default
        service: eventqueue
      record: mydata

設定HPA

在HorizontalPodAutoscaler增加一個設定如下

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 60
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 300
    scaleUp:
      policies:
      - periodSeconds: 60
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 300
  maxReplicas: 2
  metrics:
  - object:
      describedObject:
        apiVersion: v1
        kind: Service
        name: eventqueue
      metric:
        name: mydata
      target:
        type: Value
        value: 1k
    type: Object
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-container

參考資料

1. https://docs.openshift.com/container-platform/4.9/nodes/pods/nodes-pods-autoscaling.html
2. https://www.padok.fr/en/blog/scaling-prometheus-rabbitmq