Step-1: Update the domain name
Step-2: Modify the role attribute path for Grafana access
Step-3: Modify the retention, storage size, cluster name and targets based on the specific requirements
Step-4: Adjust the volume size and update the retention period accordingly
Optional: S3 bucket configuraation(Recommended for prod)
Caution: Use the sub claim instead of aud when setting up Web Identity (OIDC) IAM roles to ensure correct identity matching.
Step-4a: Create an AWS Web Identity (OIDC) IAM role with the following policy.
Copy {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AccessToLokiBucket",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<s3-bucket>",
"arn:aws:s3:::<s3-bucket>/*"
]
}
]
}
Step-4b: Update S3 details & role ARN in the below config.
Copy # deploy-as-code/helm/environments/egov-demo.yaml
loki:
persistence:
enabled: true
accessModes:
- ReadWriteOnce
size: 10Gi
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: <s3-role-arn> ## AWS arn for s3 role
additionalConfigs:
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: s3 ## AWS s3 as storage
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/index
cache_location: /data/loki/index_cache
shared_store: s3 ## AWS s3 as storage
cache_ttl: 24h
aws:
s3: s3://<region>/<s3-bucket> ## s3 region & bucket
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: s3 ## AWS s3 as storage
retention_enabled: true
compaction_interval: 168h ## compaction in hours
table_manager:
retention_deletes_enabled: true
retention_period: 168h ## retention in hours
Step-5: Make the required changes in the env-secrets file
Changes to the Alertmanager configuration in the env-secrets.yaml file.
Step-6: OAuth app configuration
Step-7: Authentication configuration for Grafana in env-secrets.yaml
Sample Env-Secrets File
Copy cluster-configs:
secrets:
db:
username: postgres
password: test123
flywayUsername: postgres
flywayPassword: test123
egov-filestore:
awskey: jdfbjdfjvnbvdk
awssecretkey: bxjcsvbvncajsb
user:
username: admin
password: demo
egov-enc-service:
master_password: demo
master_salt: q7.fr.cr
master_initialvector: 9J&asfgrU-H2
egov-notification-sms:
username: demo
password: demo
egov-pg-service:
axis_merchant_accesscode: demo
axis_merchant_id: demo
axis_merchant_pwd: demo
axis_merchant_secretkey: demo
axis_merchant_user: demo
payu_merchant_key: demo
payu_merchant_salt: demo
egov-notification-mail:
mailsenderusername: demo@demo
mailsenderpassword: demo
egov-location:
gmapskey: jbsdbvxvcmbsmnx
kafka:
clusterID: HshRPdVrcvxWoB4kuTdEbawtq
elasticsearch:
password: <Password>
oauth2:
clientSecret: <Client Sec ID>
clientID: <Client ID>
grafana:
clientID: <OAuth-key> ##change ID
clientSecret: <OAuth-token> #change secrets key
git-sync:
ssh: |-
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAxMn4y2irJKb/dAQr4FZtiBbX+VfgeNWDO6Ure90CA+f5QcjL
i0SHAbpvZ+PAPwZcYZMiOE7hdRh1xSiY7u1GQPcm1ZIboKSYahafq41XFzYGG3hk
6GHC0RPPGhW4TQ8PWbUiReSndn2VE/VY+3DItS6kSwKazqfBWVqXZ9fkAQ0yUFT1
M1rsdIZoeei9NH48UyDD4U2x/BEHMneElQbibwDuBrN/6DSzlIcgOMhePf272Nsf
GOL/SA2YJx6k7gDHjEDZ/pz6MT9XjVcDjP9y8f2udObrzIopv3C0jRZp+rKM6PFU
sLCtJSRoohmmYlexixhMFS/kAPP6Q8VyHeUcWwIDAQABAoIBAQCBZiW460yOP1l+
mjeXvn0rnYnKpaQvEIbIs6VSP1NR6jmWrkhZfWghFMyozbPePXqFltBLomLSMpFO
YZGemls14M6iZP7RtSmbqOC5V6lK0/VUHuiLfa0y+gmWp22XDi4T2O1+dApB+fYL
N6uZOuJfcRoLUN0mwlx7OvyQBgAhR7r0eePcV+yvk35qSVlKw9KreAytvE9fmLcZ
pH4jKSOFibAsDYYz9oVnwo5+aVvYl3oU2TyLwQFKmkZyKJsMOWtGw4+MVRO5/xre
WzuR8QNY/z7/A2MNQlO4KjEqkv4m/z6lh9WaDXO+PbCRRARbFcS4ZUJgXgPhtFz3
wiyXxExxAoGBAOP/GKktpUVMqtE3sXlJ4xS2l7e9w1t9kyceElXZl2FSJkqJe9Bt
PJ9FdjbB4wlxF985PkOByvOQwsGMcuMOF+BlHW/KcA2LR2vsoBY6zGTJB35YqD2V
lpdI3az0RugrYTCi3pHq/GVAc9h+V9S4+SvsuIZfrfXV+OwkeFh6gmG9AoGBANz1
nbgdQk5ZIJJvr5Y0Hn2fTKGyvHsiu/MNL5axaYxD1BUAdPdW9x7nsI42ayIzERGr
lLkO4YF4kC52LZj/wo3UlYq8ERMyH6tLnD4j/aFy4bqYAco89H43DBDAq59DTcM6
2tF+VzTNaANI4bvOTnjMTmLEJ4zRDUnb9vkAX3v3AoGAQtjVUyz16waagrMQjt4x
/S23+ABkWdvMnEh92bvtXXRnk60Rpz+P6abFDTL1rRwCgslWzxYr+hO0dmkGejn0
mC8tXUx+ZAo1C5iaK0pcCSTD1LCLy1qjh4GutPn+HC4z1b27Ag9ipxEppg0NFWqS
a+WBCKze5VgyHpJm0pJAzgUCgYASs6tMyRUyonKSUmevM+wcv93xlbpERdVYphYQ
ECYZ3CfYOzirMq4p7HxSHSMGOwJH15j37N2DYtv5QsFrQMKL1KFvo6liUYzCp9yq
mcs+3gVjELieEHi1Mh2QUW51RXIQgyvALYxeCMCz/ng0uCqGKOy9iVK7pXoVdUu7
GZ/7UwKBgQDcgSKcRk17AZ5w6cTGV+POpTnHCwq8cC37t8YBRLBqXgbQObVGViYp
D+t2DZeZ22VQCXbyBE8NTMi/9c+Zo+uguAe8tzroxhAP9uzsHw6qTb7QHQHZ/CPK
wBkBi92ZelIXkby0L8ljdQKEDbhPc8MBpinoIQgJurbZvdvS9zhjuljLssw==
-----END RSA PRIVATE KEY-----
known_hosts: github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDsskzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=
alertmanager:
config:
global:
slack_api_url: https://hooks.slack.com ##change the slack api url
resolve_timeout: 5m
route:
group_by:
- alertname
group_wait: 30s
receiver: slack-notification
group_interval: 5m
repeat_interval: 10m
routes:
- receiver: slack-notification
match_re:
severity: warning|critical
continue: true
- receiver: email-notification
match:
severity: critical
receivers:
- name: slack-notification
slack_configs:
- channel: '<slack-channel>' ##change the slack channel name
send_resolved: true
username: Alertmanager
title: |
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
text: |-
{{ range .Alerts -}}
{{- "\n" -}}
*Alert:* {{ .Annotations.summary }}
{{ if .Labels.severity }}*Severity:* `{{ .Labels.severity }}`{{ end }}
*Cluster:* {{ .Labels.cluster }}
*Details:*
{{ .Annotations.description }}
{{ end }}
color: |-
{{ if eq .Status "firing" -}}
{{ if eq .CommonLabels.severity "warning" -}}
warning
{{- else if eq .CommonLabels.severity "critical" -}}
danger
{{- else -}}
#439FE0
{{- end -}}
{{ else -}}
good
{{- end }}
- name: email-notification
email_configs:
- to: <Email ID> ##change the Email ID to get the alert in the Email
from: <Email ID>
smarthost: smtp.gmail.com:587
auth_username: <Email ID> # Ex: unified.alerts@egovernments.org
auth_password: <Password> # Ex: mujp cgjj fhdv wieu
send_resolved: true
headers:
subject: |
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.cluster }} - {{ .CommonLabels.alertname }}
html: |-
<html>
<head>
<title>Alert!</title>
</head>
<body>
{{ range .Alerts }}
<ul>
<li> <b>Alert Name:</b> {{ .Labels.alertname }} </li>
<li> <b>Severity:</b> {{ if eq .Labels.severity "critical" }}<b style="color:red;">CRITICAL</b>{{ else if eq .Labels.severity "warning" }}<b style="color:orange;">WARNING</b>{{ else }}<b>{{ .Labels.severity | toUpper }}</b>{{ end }} </li>
<li> <b>Summary:-</b> {{ .Annotations.summary }} </li>
<li> <b>Cluster:-</b> {{ .Labels.cluster }} </li>
<li> <b>Details:</b>
<p style="margin-left: 20px; white-space: pre-wrap;"> {{ .Annotations.description }} </p>
</li>
</ul><br>
{{ end }}
</body></html>
Follow the below steps to create a KMS key and configure SOPs for encryption and decryption.
1. Create IAM User & Attach Policies
Go to AWS Management Console.
Go to IAM and click Policies on the left-hand side of the toolbar.
Click Create Policy and then press Next. Click on JSON add the below policy and click Next.
Copy {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCreateRole",
"Effect": "Allow",
"Action": "iam:CreateRole",
"Resource": "arn:aws:iam::349271159511:role/*"
}
]
} Give the name to the policy and Create Policy.
Now click on Users in the console on the left side.
Click Add User, provide the name of the specific user and click Next.
In permission options click Attach Policies directly. Select Administration Access, AWSKeyManagementServicePoweruser and also attach the policy which you have created in the previous steps and then click Next.
Verify the name of the user and the 3 policies attached or not and then click Create User.
2. Create KMS Key
Go to AWS Management Console.
Go to KMS and Custom Managed Keys. Click Create Key.
In Configure Key, use the default options and then click Next. Provide Name in alias and the give the administrator access to the IAM user created in the previous step. Select the users to give permissions to encrypt and decrypt using this key and then click Next.
Attach the below policy in the key policy and then click finish. Make sure to provide the IAM user you created in the below-highlighted placeholder.
Copy {
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::349271159511:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::349271159511:role/aws-reserved/sso.amazonaws.com/ap-south-1/AWSReservedSSO_AdministratorAccess_3b9b4bb9eebf66ac"
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::349271159511:user/<IAM USER>"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:ap-south-1:349271159511:key/29adbf26-7b85-4469-8c9e-f8050fd19a8e"
},
{
"Sid": "Allow attachment of persistent resources",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::349271159511:role/aws-reserved/sso.amazonaws.com/ap-south-1/AWSReservedSSO_AdministratorAccess_3b9b4bb9eebf66ac"
},
"Action": [
"kms:CreateGrant",
"kms:ListGrants",
"kms:RevokeGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
]
} Copy the arn value after creating the KMS Key.
3. Placing the KMS arn value in the deployment manifest file
Go to the below code and add the arn key in .sops.yaml file.
Copy DIGIT-DevOps/blob/DIGIT-2.9LTS-monitoring/deploy-as-code/charts/.sops.yaml Next, cd to deploy-as-code and run the below command.
Copy sops --encrypt --in-place charts/env-secrets.yaml Now to see the encrypted secrets. We can decrypt the secrets using the below command.
Copy sops -d environments/env-secrets.yaml 
