k8s-安装Grafana/展示[node|docker|k8s]资源指标

k8s-安装Grafana/展示[node|docker|k8s]资源指标

Grafana是一个跨平台的开源的度量分析和可视化工具，可以将采集的数据可视化的展示，并及时通知给告警接收方。它主要有以下六大特点：

• 1、展示方式：快速灵活的客户端图表，面板插件有许多不同方式的可视化指标和日志，官方库中具有丰富的仪表盘插件，比如热图、折线图、图表等多种展示方式；
• 2、数据源：Graphite，InfluxDB，OpenTSDB，Prometheus，Elasticsearch，CloudWatch和KairosDB等；
• 3、通知提醒：以可视方式定义最重要指标的警报规则，Grafana将不断计算并发送通知，在数据达到阈值时通过Slack、PagerDuty等获得通知；
• 4、混合展示：在同一图表中混合使用不同的数据源，可以基于每个查询指定数据源，甚至自定义数据源；
• 5、注释：使用来自不同数据源的丰富事件注释图表，将鼠标悬停在事件上会显示完整的事件元数据和标记。

安装Grafana

node节点部署

[root@localhost ~]# docker pull angelnu/heapster-grafana:v5.0.4-amd64

更新yaml文件：

[root@master01 ~]#  kubectl apply -f grafana.yaml
deployment.apps/monitoring-grafana created
service/monitoring-grafana created

#查看grafana是否创建成功：

[root@master01 ~]# kubectl get pods -n kube-system -l task=monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
monitoring-grafana-598d4ffdbc-zb2pn   1/1     Running   0          21s

grafana.yaml文件内容如下：

[root@master01 ~]# cat grafana.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      task: monitoring
      k8s-app: grafana
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: angelnu/heapster-grafana:v5.0.4-amd64
        ports:
        - containerPort: 3000
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certificates
          readOnly: true
        - mountPath: /var
          name: grafana-storage
        env:
        - name: INFLUXDB_HOST
          value: monitoring-influxdb
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
          # The following env variables are required to make Grafana accessible via
          # the kubernetes api-server proxy. On production clusters, we recommend
          # removing these env variables, setup auth for grafana, and expose the grafana
          # service using a LoadBalancer or a public IP.
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: Admin
        - name: GF_SERVER_ROOT_URL
          # If you're only using the API Server proxy, set this value instead:
          # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
          value: /
      volumes:
      - name: ca-certificates
        hostPath:
          path: /etc/ssl/certs
      - name: grafana-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-grafana
  name: monitoring-grafana
  namespace: kube-system
spec:
  # In a production setup, we recommend accessing Grafana through an external Loadbalancer
  # or through a public IP.
  # type: LoadBalancer
  # You could also use NodePort to expose the service at a randomly-generated port
  # type: NodePort
  ports:
  - port: 80
    targetPort: 3000
    nodePort: 30003
  selector:
    k8s-app: grafana
  type: NodePort

查看grafana前端的service

[root@master01 ~]# kubectl get svc -n kube-system | grep grafan
monitoring-grafana   NodePort    10.1.28.97   <none>        80:30003/TCP             74s

1）登陆grafana，在浏览器访问

192.168.1.180:30003

2）配置grafana界面：

开始配置grafana的web界面：
选择Create your first data source

出现如下

Name: Prometheus

Type: Prometheus

HTTP 处的URL写 如下：

http://prometheus.monitor-sa.svc:9090

配置好的整体页面如下：

点击左下角Save & Test，出现如下Data source is working，说明prometheus数据源成功的被grafana接入了

导入的监控模板，可在如下链接搜索

https://grafana.com/dashboards?dataSource=prometheus&search=kubernetes

可直接导入node_exporter.json监控模板，这个可以把node节点指标显示出来

没有数据排查方法：

• grafana里编辑对应没有数据的指标，找到指标参数粘贴到prometheus里，看有没有对应数据，有可能是版本不一样指标表换了，就按照这种方法找到正确的指标替换即可

grafana展示node资源指标

怎么导入监控模板，按如下步骤：

上面Save & Test测试没问题之后，就可以返回Grafana主页面

点击左侧+号下面的Import，出现如下界面

#导入魔板后展示效果

grafana展示docker 容器资源指标

导入docker_rev1.json监控模板，步骤和上面导入node_exporter.json步骤一样，导入之后显示如下：

grafana展示k8s 组件资源指标

安装kube-state-metrics组件

kube-state-metrics是什么？

kube-state-metrics通过监听API Server生成有关资源对象的状态指标，比如Deployment、Node、Pod，需要注意的是kube-state-metrics只是简单的提供一个metrics数据，并不会存储这些指标数据，所以我们可以使用Prometheus来抓取这些数据然后存储，主要关注的是业务相关的一些元数据，比如Deployment、Pod、副本状态等；调度了多少个replicas？现在可用的有几个？多少个Pod是running/stopped/terminated状态？Pod重启了多少次？我有多少job在运行中。

1）创建sa，并对sa授权

通过kubectl apply更新资源清单yaml文件

[root@master01 ~]# kubectl apply -f kube-state-metrics-rbac.yaml 
serviceaccount/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created

kube-state-metrics-rbac.yaml文件内容如下：

[root@master01 ~]# cat kube-state-metrics-rbac.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources: ["nodes", "pods", "services", "resourcequotas", "replicationcontrollers", "limitranges", "persistentvolumeclaims", "persistentvolumes", "namespaces", "endpoints"]
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources: ["daemonsets", "deployments", "replicasets"]
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

2)安装kube-state-metrics组件

下载镜像

[root@node01 ~]# docker pull kubesphere/kube-state-metrics:v1.9.7

通过kubectl apply更新yaml文件

[root@master01 prometheus]# kubectl apply -f kube-state-metrics-deploy.yaml 

kube-state-metrics-deploy.yaml文件内容如下：

[root@master01 prometheus]# cat kube-state-metrics-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: kubesphere/kube-state-metrics:v1.9.7
        ports:
        - containerPort: 8080

查看kube-state-metrics是否部署成功

[root@master01 prometheus]# kubectl get pods -n kube-system -l app=kube-state-metrics
NAME                                  READY   STATUS    RESTARTS   AGE
kube-state-metrics-6c6c5b9c88-88fbx   1/1     Running   0          108s

3）创建service

[root@master01 prometheus]# kubectl apply -f kube-state-metrics-svc.yaml
service/kube-state-metrics created

kube-state-metrics-svc.yaml文件内容如下：

[root@master01 prometheus]# cat kube-state-metrics-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  name: kube-state-metrics
  namespace: kube-system
  labels:
    app: kube-state-metrics
spec:
  ports:
  - name: kube-state-metrics
    port: 8080
    protocol: TCP
  selector:
    app: kube-state-metrics

在grafana web界面导入Kubernetes Cluster (Prometheus)-1577674936972.json和Kubernetes cluster monitoring (via Prometheus) (k8s 1.16)-1577691996738.json，Kubernetes Cluster (Prometheus)-1577674936972.json和Kubernetes cluster monitoring (via Prometheus) (k8s 1.16)-1577691996738.json