eGov Monitoring & Alerting Setup

This doc will cover how you can set up the monitoring and alerting on existing k8s cluster either with help of go lang script or Jenkins deployment Jobs.

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud

Pre-reads

https://prometheus.io/docs/introduction/overview OAuth2-Proxy Setup

Pre-requisites

• DIGIT uses golang (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - Linux or Windows or Mac

• All DIGIT services are packaged using helm charts Installing Helm

• kubectl is a CLI to connect to the kubernetes cluster from your machine

• Install Visualstudio IDE Code for better code/configuration editing capabilities

• Git

• OAuth2-Proxy Setup

Architectur

prometheus-operator chart includes multiple components and is suitable for a variety of use-cases.

The default installation is intended to suit monitoring a kubernetes cluster the chart is deployed onto. It closely matches the kube-prometheus project.

• prometheus-operator

• prometheus

• alertmanager

• node-exporter

• kube-state-metrics

• grafana

• service monitors to scrape internal kubernetes components

  • kube-apiserver

  • kube-scheduler

  • kube-controller-manager

  • etcd

  • kube-dns/coredns

  • kube-proxy

With the installation, the chart also includes dashboards and alerts.

Deployment steps:

1. Add the below grafana init container parameters to your env config file

   1. Chose your env config file, if you are deploying monitoring and alerting into the qa environment chose qa.yaml similarly for uat, dev, and other environments.

grafana:
  initContainers:
    gitSync:
      enabled: true
      repo: "[e-mail]:egovernments/configs" #REPLACE with your configs repo
      branch: "<branch_name>" #REPLACE with config repo branch name
  dashboardsFolder: /work-dir/configs/monitoring-dashboards    

Depending upon your environment config file update the configs repo branch (like for qa.yaml add qa branch and uat.yaml it would be UAT the branch)

2. Add monitoring-dashboards folder to the configs repo's branch which you selected in 1st step.

3. Enable the serviceMonitor in the nginx-ingress configs which are available in the same <env>.yaml and redeploy the nginx-ingress.

nginx-ingress:
  replicas: 1
  default-backend-service: "egov/nginx"
  namespace: egov
  cert-issuer: "letsencrypt-prod"
  ssl-protocols: "TLSv1.2 TLSv1.3"
  ssl-ciphers: "EECDH+CHACHA20:EECDH+AES"
  ssl-ecdh-curve: "X25519:prime256v1:secp521r1:secp384r1"
  controller:
    image:
      repository: egovio/nginx-ingress-controller
      tag: "0.26.1"     
    metrics:            #To collect the matrics data from nginx-ingress.
      enabled: true
      serviceMonitor:   #To enable the service monitoring of nginx-ingress
        enabled: true
    service:
      prometheusRule:
        enabled: true

go run main.go deploy -e <environment_name> -c 'nginx-ingress'

4. To enable alerting, Add alertmanager secret in <env>-secrets.yaml

If you want you can change the slack channel and other details like group_wait, group_interval, and repeat_interval according to your values.

5. You can deploy the prometheus-operator using one of the below methods.

1. Deploy using go lang deployer

go run main.go deploy -e <environment_name> -c 'prometheus-operator,grafana,prometheus-kafka-exporter'

2. Deploy using Jenkin’s deployment job. (here we are using deploy-to-dev, you can choose your environment specific deployment job )

You can connect to the monitoring console at https://<your_domin_name>/monitoring/

To create a new panel in the existing dashboard:-

1. Login to the dashboard and click on add panel

1. Set all required queries and apply the changes. Export the JSON file by clicking on the save dashboard

3. Go to the configs repo and select your branch. In the branch look for the monitoring-dashboards folder and update the existing *-dashboard.json with a newly exported JSON file.