K8s 部署 Prometheus + Grafana

一、简介

1. prometheus

一款开源的监控&报警&时间序列数据库的组合，起始是由 soundcloud 公司开发的基本原理是通过 http 协议周期性抓取被监控组件的状态，这样做的好处是任意组件只要提供 http 接口就可以接入监控系统，不需要任何 sdk 或者其他的集成过程。这样做非常适合虚拟化环境比如 vm 或者 docker输出被监控组件信息的 http 接口被叫做 exporter 。目前互联网公司常用的组件大部分都有 exporter 可以直接使用，比如 varnish、haproxy、nginx、mysql、linux 系统信息（包括磁盘、内存、cpu、网络等），具体支持的源看：https://github.com/prometheus特点：一个多维数据模型（时间序列由指标名称定义和设置键/值尺寸）非常高效的存储，平均一个采样数据占 ~3.5bytes 左右，320 万的时间序列，每 30 秒采样，保持 60 天，消耗磁盘大概 228g一种灵活的查询语言不依赖分布式存储，单个服务器节点时间集合通过 http 上的 pull 模型进行通过中间网关支持推送时间通过服务发现或静态配置发现目标多种模式的图形和仪表板支持

2. grafana

一个跨平台的开源的度量分析和可视化工具，可以通过将采集的数据查询然后可视化的展示，并及时通知特点：展示方式：快速灵活的客户端图表，面板插件有许多不同方式的可视化指标和日志，官方库中具有丰富的仪表盘插件，如热图、折线图、图表等多种展示方式数据源：graphite，influxdb，opentsdb，prometheus，elasticarch，cloudwatch 和 kairosdb 等通知提醒：以可视方式定义最重要指标的警报规则，grafana 将不断计算并发送通知，在数据达到阈值时通过 slack、pagerduty 等获得通知混合展示：在同一图表中混合使用不同的数据源，可以基于每个查询指定数据源，甚至自定义数据源注释：使用来自不同数据源的丰富事件注释图表，将鼠标悬停在事件上会显示完整的事件元数据和标记过滤器：ad-hoc 过滤器允许动态创建新的键/值过滤器，这些过滤器会自动应用于使用该数据源的所有查询

3. 效果展示

二、部署

$ kubectl create ns ns-monitor$ kubectl create -f ...$ kubectl get all -n ns-monitorname                              ready   status    restarts   agepod/node-exporter-rcbss           1/1     running   0          4h41mpod/grafana-5567c66c9d-49b5w      1/1     running   0          4h25mpod/prometheus-5ccc8db98f-lkwf5   1/1     running   0          3h12mname                            type       cluster-ip      external-ip   port(s)          agervice/node-exporter-rvice   nodeport   10.43.75.152    <none>        9100:31672/tcp   4h41mrvice/grafana-rvice         nodeport   10.43.26.238    <none>        3000:32534/tcp   4h25mrvice/prometheus-rvice      nodeport   10.43.174.110   <none>        9090:31396/tcp   3h12m

grafana 和 prometheus 没有配置nodeport，端口随机生成

1. node-exporter

用于采集 k8s 集群中各个节点的物理指标，如 memory、cpu 等。可以直接在每个物理节点直接安装

kind: daemontapiversion: apps/v1metadata:   labels:    app: node-exporter  name: node-exporter  namespace: ns-monitorspec:  revisionhistorylimit: 10  lector:    matchlabels:      app: node-exporter  template:    metadata:      labels:        app: node-exporter    spec:      containers:        - name: node-exporter          image: prom/node-exporter:v0.16.0          ports:            - containerport: 9100              protocol: tcp              name:http      hostnetwork: true  # 获得node的物理指标信息      hostpid: true  # 获得node的物理指标信息#      tolerations:  # master节点#        - effect: noschedule#          operator: exists---kind: rviceapiversion: v1metadata:  labels:    app: node-exporter  name: node-exporter-rvice  namespace: ns-monitorspec:  ports:    - name:http      port: 9100      nodeport: 31672      protocol: tcp  type: nodeport  lector:    app: node-exporter

2. prometheus

apiversion: rbac.authorization.k8s.io/v1kind: clusterrolemetadata:name: prometheusrules:- apigroups: [""]  # "" indicates the core api groupresources:- nodes- nodes/proxy- rvices- endpoints- podsverbs:- get- watch- list- apigroups:- extensionsresources:- ingressverbs:- get- watch- list- nonresourceurls: ["/metrics"]verbs:- get---apiversion: v1kind: rviceaccountmetadata:name: prometheusnamespace: ns-monitorlabels:app: prometheus---apiversion: rbac.authorization.k8s.io/v1kind: clusterrolebindingmetadata:name: prometheussubjects:- kind: rviceaccountname: prometheusnamespace: ns-monitorroleref:kind: clusterrolename: prometheusapigroup: rbac.authorization.k8s.io---apiversion: v1kind: configmapmetadata:name: prometheus-confnamespace: ns-monitorlabels:app: prometheusdata:prometheus.yml: |-# my global configglobal:scrape_interval:     15s  # t the scrape interval to every 15 conds. default is every 1 minute.evaluation_interval: 15s  # evaluate rules every 15 conds. the default is every 1 minute.# scrape_timeout is t to the global default (10s).# alertmanager configurationalerting:alertmanagers:- static_configs:- targets:# - alertmanager:9093# load rules once and periodically evaluate them according to the global 'evaluation_interval'.rule_files:# - "first_rules.yml"# - "cond_rules.yml"# a scrape configuration containing exactly one endpoint to scrape:# here it's prometheus itlf.scrape_configs:# the job name is added as a label `job=<job_name>` to any timeries scraped from this config.- job_name: 'prometheus'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['localhost:9090']- job_name: 'grafana'static_configs:- targets:- 'grafana-rvice.ns-monitor:3000'- job_name: 'kubernetes-apirvers'kubernetes_sd_configs:- role: endpoints# default to scraping over https. if required, just disable this or change to# `http`.scheme: https# this tls & bearer token file config is ud to connect to the actual scrape# endpoints for cluster components. this is parate to discovery auth# configuration becau discovery & scraping are two parate concerns in# prometheus. the discovery auth config is automatic if prometheus runs inside# the cluster. otherwi, more config options have to be provided within the# <kubernetes_sd_config>.tls_config:ca_file: /var/run/crets/kubernetes.io/rviceaccount/ca.crt# if your node certificates are lf-signed or u a different ca to the# master ca, then disable certificate verification below. note that# certificate verification is an integral part of a cure infrastructure# so this should only be disabled in a controlled environment. you can# disable certificate verification by uncommenting the line below.## incure_skip_verify: truebearer_token_file: /var/run/crets/kubernetes.io/rviceaccount/token# keep only the default/kubernetes rvice en我的老妈dpoints for the https port. this# will add targets for each api rver which kubernetes adds an endpoint to# the default/kubernetes rvice.relabel_configs:- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_rvice_name, __meta_kubernetes_endpoint_port_name]action: keepregex: default;kubernetes;https# scrape config for nodes (kubelet).## rather than connecting directly to the node, the scrape is proxied though the# kubernetes apirver.  this means it will work if prometheus is running out of# cluster, or can't connect to nodes for some other reason (e.g. becau of# firewalling).- job_name: 'kubernetes-nodes'# default to scraping over https. if required, just disable this or change to# `http`.scheme: https# this tls & bearer token file config is ud to connect to the actual scrape# endpoints for cluster components. this is parate to discovery auth# configuration becau discovery & scraping are two parate concerns in# prometheus. the discovery auth config is automatic if prometheus runs inside# the cluster. otherwi, more config options have to be provided within the# <kubernetes_sd_config>.tls_config:ca_file: /var/run/crets/kubernetes.io/rviceaccount/ca.crtbearer_token_file: /var/run/crets/kubernetes婚礼策划人.io/rviceaccount/tokenkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- target_label: __address__replacement: kubernetes.default.svc:443- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/${1}/proxy/metrics# scrape config for kubelet cadvisor.## this is required for kubernetes 1.7.3 and later, where cadvisor metrics# (tho who names begin with 'container_') have been removed from the# kubelet metrics endpoint.  this job scrapes the cadvisor endpoint to# retrieve tho metrics.## in kubernetes 1.7.0-1.7.2, the metrics are only expod on the cadvisor# http endpoint; u "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"# in that ca (and ensure cadvisor's http rver hasn't been disabled with# the --cadvisor-port=0 kubelet flag).## this job is not necessary and should be removed in kubernetes 1.6 and# earlier versions, or it will cau the metrics to be scraped twice.- job_name: 'kubernetes-cadvisor'# default to scraping over https. if required, just disable this or change to# `http`.scheme: https# this tls & bearer token file config is ud to connect to the actual scrape# endpoints for cluster components. this is parate to 英雄联盟怎么回复私聊discovery auth# configuration becau discovery & scraping are two parate concerns in# prometheus. the discovery auth config is automatic if prometheus runs inside# the cluster. otherwi, more config options have to be provided within the# <kubernetes_sd_config>.tls_config:ca_file: /var/run/crets/kubernetes.io/rviceaccount/ca.crtbearer_token_file: /var/run/crets/kubernetes.io/rviceaccount/tokenkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- target_label: __address__replacement: kubernetes.default.svc:443- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor# scrape config for rvice endpoints.## the relabeling allows the actual rvice scrape endpoint to be configured# via the following annotations:## * `prometheus.io/scrape`: only scrape rvices that have a value of `true`# * `prometheus.io/scheme`: if the metrics endpoint is cured then you will need# to t this to `https` & most likely t the `tls_config` of the scrape config.# * `prometheus.io/path`: if the metrics path is not `/metrics` override this.# * `prometheus.io/port`: if the metrics are expod on a different port to the# rvice then t this appropriately.- job_name: 'kubernetes-rvice-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_rvice_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_rvice_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_rvice_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_rvice_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_rvice_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_rvice_name]action: replacetarget_label: kubernetes_name# example scrape config for probing rvices via the blackbox exporter.## the relabeling allows the actual rvice scrape endpoint to be configured# via the following annotations:## * `prometheus.io/probe`: only probe rvices that have a value of `true`- job_name: 'kubernetes-rvices'metrics_path: /probeparams:module: [http_2xx]kubernetes_sd_configs:- role: rvicerelabel_configs:- source_labels: [__meta_kubernetes_rvice_annotation_prometheus_io_probe]action: keepregex: true- source_labels: [__address__]target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_rvice_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: kubernetes_namespace- source_labels: [__meta_kubernetes_rvice_name]target_label: kubernetes_name# example scrape config for probing ingress via the blackbox exporter.## the relabeling allows the actual ingress scrape endpoint to be configured# via the following annotations:## * `prometheus.io/probe`: only probe rvices that have a value of `true`- job_name: 'kubernetes-ingress'metrics_path: /probeparams:module: [http_2xx]kubernetes_sd_configs:- role: ingressrelabel_configs:- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]action: keepregex: true- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]regex: (.+);(.+);(.+)replacement: ${1}://${2}${3}target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_ingress_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: kubernetes_namespace- source_labels: [__meta_kubernetes_ingress_name]target_label: kubernetes_name# example scrape config for pods## the relabeling allows the actual pod scrape endpoint to be configured via the# following annotations:## * `prometheus.io/scrape`: only scrape pods that have a value of `true`# * `prometheus.io/path`: if thhis什么意思e metrics path is not `/metrics` override this.# * `prometheus.io/port`: scrape the pod on the indicated port instead of the# pod's declared ports (default is a port-free target if none are declared).- job_name: 'kubernetes-pods'kubernetes_sd_configs:- role: podrelabel_configs:- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]action: replaceregex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2target_label: __address__- action: labelmapregex: __meta_kubernetes_pod_label_(.+)- source_labels: [__meta_kubernetes_namespa趣味小游戏ce]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_pod_name]action: replacetarget_label: kubernetes_pod_name---apiversion: v1kind: configmapmetadata:name: prometheus-rulesnamespace: ns-monitorlabels:app: prometheusdata:cpu-usage.rule: |groups:- name: nodecpuusagerules:- alert: nodecpuusageexpr: (100 - (avg by (instance) (irate(node_cpu{name="node-exporter",mode="idle"}[5m])) * 100)) > 75for: 2mlabels:verity: "page"annotations:summary: "{{$labels.instance}}: high cpu usage detected"description: "{{$labels.instance}}: cpu usage is above 75% (current value is: {{ $value }})"---apiversion: v1kind: persistentvolumemetadata:name: "prometheus-data-pv"labels:name: prometheus-data-pvrelea: stablespec:capacity:storage: 5giaccessmodes:- readwriteoncepersistentvolumereclaimpolicy: recyclenfs:path: /nfs/prometheus/datarver: 192.168.11.210---apiversion: v1kind: persistentvolumeclaimmetadata:name: prometheus-data-pvcnamespace: ns-monitorspec:accessmodes:- readwriteonceresources:requests:storage: 5gilector:matchlabels:name: prometheus-data-pvrelea: stable---kind: deploymentapiversion: apps/v1metadata:labels:app: prometheusname: prometheusnamespace: ns-monitorspec:replicas: 1revisionhistorylimit: 10lector:matchlabels:app: prometheustemplate:metadata:labels:app: prometheusspec:rviceaccountname: prometheuscuritycontext:runasur: 0containers:- name: prometheusimage: prom/prometheus:latestimagepullpolicy: ifnotprentvolumemounts:- mountpath: /prometheusname: prometheus-data-volume- mountpath: /etc/prometheus/prometheus.ymlname: prometheus-conf-volumesubpath: prometheus.yml- mountpath: /etc/prometheus/rulesname: prometheus-rules-volumeports:- containerport: 9090protocol: tcpvolumes:- name: prometheus-data-volumepersistentvolumeclaim:claimname: prometheus-data-pvc- name: prometheus-conf-volumeconfigmap:name: prometheus-conf- name: prometheus-rules-volumeconfigmap:name: prometheus-rulestolerations:- key: node-role.kubernetes.io/mastereffect: noschedule---kind: rviceapiversion: v1metadata:annotations:prometheus.io/scrape: 'true'labels:app: prometheusname: prometheus-rvicenamespace: ns-monitorspec:ports:- port: 9090targetport: 9090lector:app: prometheustype: nodeport

3. grafana

apiversion: v1kind: persistentvolumemetadata:name: "grafana-data-pv"labels:name: grafana-data-pvrelea: stablespec:capacity:storage: 5giaccessmodes:- readwriteoncepersistentvolumereclaimpolicy: recyclenfs:path: /nfs/grafana/datarver: 192.168.11.210---apiversion: v1kind: persistentvolumeclaimmetadata:name: grafana-data-pvcnamespace: ns-monitorspec:accessmodes:- readwriteonceresources:requests:storage: 5gilector:matchlabels:name: grafana-data-pvrelea: stable---kind: deploymentapiversion: apps/v1metadata:labels:app: grafananame: grafananamespace: ns-monitorspec:replicas: 1revisionhistorylimit: 10lector:matchlabels:app: grafanatemplate:metadata:labels:app: grafanaspec:curitycontext:runasur: 0containers:- name: grafanaimage: grafana/grafana:latestimagepullpolicy: ifnotprentenv:- name: gf_auth_basic_enabledvalue: "true"- name: gf_auth_anonymous_enabledvalue: "fal"readinessprobe:httpget:path: /loginport: 3000volumemounts:- mountpath: /var/lib/grafananame: grafana-data-volumeports:- containerport: 3000protocol: tcpvolumes:- name: grafana-data-volumepersistentvolumeclaim:claimname: grafana-data-pvc---kind: rviceapiversion: v1metadata:labels:app: grafananame: grafana-rvicenamespace: ns-monitorspec:ports:- port: 3000targetport: 3000lector:app: grafanatype: nodeport

配置数据源

import dashboard from file（非必须）

https://files.cnblogs.com/files/lb477/kubernetes-pod-resources.json

参考：