Central Monitoring Dashboard Setup

In this tutorial, we will go through the step by step process of setup central dashboard Monitoring

Pre-reads
​https://github.com/kubecost/cost-analyzer-helm-chart#kubecost-helm-chart
https://prometheus.io/docs/introduction/overview/
https://grafana.com/docs/


Pre-requisites
DIGIT uses golang (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - Linux or Windows or Mac​
All DIGIT services are packaged using helm charts 
Installing Helm​
​kubectl is a CLI to connect to the kubernetes cluster from your machine
​Install Visualstudio IDE Code for better code/configuration editing capabilities
Git
cost-analyzer - cost-analyzer  must be deployed on the client side 
Prometheus-operator - prometheus must be deployed on the client side 
prometheus-kafka-exporter (A Prometheus exporter acts as a proxy between such an applications and the Prometheus server) - prometheus-kafka-exporter must be deployed on the client side 
Step 1: Install the Prometheus Operator on each client cluster
eGov Monitoring & Alerting Setup
CORE
Step 2: Exposes the prometheus Operator 
exposes the  prometheus Operator using nginx-ingress rule in each client cluster. This makes it easy to access Prometheus metrics in central-dashboard clusters
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod  //By using this annotation, a certificate will be issued
  labels:
    app: prometheus-operator
  name: prometheus-prometheus
  namespace: monitoring
spec:
  rules:
  - host: <host_name>                     //Replace with prometheus host
    http:
      paths:
      - backend:
          service:
            name: prometheus-prometheus
            port:
              number: 9090
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - <host_name>                         //Replace with prometheus host
    secretName: <host_name>-tls-certs     //Replace with prometheus host
Note: Ensure you create the CANAME DNS record with the hostname and loadbalancer ID.
Step 3: Install cost-analyzer on each client cluster
Kubecost provides the visibility into current and historical Kubernetes spend and resource allocation. These provide cost transparency in Kubernetes environments.
You can deploy the cost-analyzer using one of the below methods.
     1. Deploy using go lang deployer
git clone https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/deploy-as-code/egov-deployer
go run main.go -e <environment_name> "cost-analyzer"
    2.  Deploy using Jenkin’s deployment job. (here we are using deploy-to-dev, you can choose your environment specific deployment job )
Step 4: Install Grafana on the central-dashboard cluster
Below grafana configuration should be added to the environments file, and then Grafana should be deployed using one of the following methods.
Based on the number of client clusters, you have to add datasources. There should be one entry per client cluster as shown below
grafana: 
  image:
    repository: grafana/grafana
    tag: 9.0.0
​
  initContainers:
    gitSync:
      enabled: true
      repo: "[email protected]:egovernments/configs"  // repo contain the dashboard json files 
      branch: "central-dashboard"                 
  ingress:
    hostName: central-dashboard.digit.org  //central-dashboard hostname
    context: ""
    additionalAnnotations: | 
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s"
      serve_from_sub_path: true 
​
  env: |
    - name: GF_SERVER_DOMAIN
      value: {{ .Values.ingress.hostName | quote }}    
  datasources:                // Configure Grafana to use prometheus datasources
    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: DIGIT-Dev
        type: prometheus
        url: https://<Dev_hostname>  //Add exposed hostname for dev environment to access the dev prometheus metrics
        isDefault: false 
      - name: DIGIT-QA
        type: prometheus
        url: https://<QA_hostname>
        isDefault: false
      - name: DIGIT-UAT
        type: prometheus
        url: https://<UAT_hostname>
        isDefault: false
      - name: DIGIT-Staging
        type: prometheus
        url: https://<Staging_hostname>   
        isDefault: false
1.Deploy using go lang deployer
git clone https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/deploy-as-code/egov-deployer
go run main.go -e <environment_name> "grafana"
​
2.Deploy using Jenkin’s deployment job.
​
Step 4: DNS Mapping
To access the monitoring central dashboard with this https://central-dashboard.digit.org url. Ensure you create the CANAME DNS record with the hostname and loadbalancer ID.

Setup Central Instance Infra

Kubernetes

Last modified 8mo ago