eGov Monitoring & Alerting Setup
This doc will cover how you can set up the monitoring and alerting on existing k8s cluster either with help of go lang script or Jenkins deployment Jobs.
Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud
The default installation is intended to suit monitoring a kubernetes cluster the chart is deployed onto. It closely matches the kube-prometheus project.
- service monitors to scrape internal kubernetes components
With the installation, the chart also includes dashboards and alerts.
- 1.Add the below grafana init container parameters to your env config file
- 1.Chose your env config file, if you are deploying monitoring and alerting into the qa environment chose qa.yaml similarly for uat, dev, and other environments.
repo: "[email protected]:egovernments/configs" #REPLACE with your configs repo
branch: "<branch_name>" #REPLACE with config repo branch name
Depending upon your environment config file update the configs repo branch (like for qa.yaml add qa branch and uat.yaml it would be UAT the branch)
3. Enable the serviceMonitor in the nginx-ingress configs which are available in the same <env>.yaml and redeploy the nginx-ingress.
ssl-protocols: "TLSv1.2 TLSv1.3"
metrics: #To collect the matrics data from nginx-ingress.
serviceMonitor: #To enable the service monitoring of nginx-ingress
go run main.go deploy -e <environment_name> -c 'nginx-ingress'
4. To enable alerting, Add alertmanager secret in <env>-secrets.yaml
If you want you can change the slack channel and other details like group_wait, group_interval, and repeat_interval according to your values.
5. You can deploy the prometheus-operator using one of the below methods.
1. Deploy using go lang deployer
go run main.go deploy -e <environment_name> -c 'prometheus-operator,grafana,prometheus-kafka-exporter'
2. Deploy using Jenkin’s deployment job. (here we are using deploy-to-dev, you can choose your environment specific deployment job )
- 1.Login to the dashboard and click on add panel
- 1.Set all required queries and apply the changes. Export the JSON file by clicking on the save dashboard
3. Go to the configs repo and select your branch. In the branch look for the monitoring-dashboards folder and update the existing *-dashboard.json with a newly exported JSON file.