DIGIT Core
Master
UrbanSanitationPublic FinanceHealthAcademyFeedback
Search
⌃K
Links

1. Choose Infra

Steps to choose and setup the required infrastructure/environment for deployment

Topics Covered:

Setting Up Kubernetes Cluster On A Single Machine/VM

Click on the relevant environment option (local machine/VM/Cloud AWS)below to start with the setup. Before proceeding with the setup, make sure to check the hardware requirement of the selected environment.
On local machine/VM
Pre-requisites
  • Linux distribution running in a VM or bare metal
    • Install curl, wget git, and tar (if they're not already installed):
      • sudo apt-get install curl git wget tar
    • Install Docker​
    • ​Install kubectl on Linux
    • Open terminal and Install k3d(v4.4.8) on Linux using the below command
      • wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v4.4.8 bash
  • OSX or Mac
    • ​Docker Desktop local Kubernetes cluster enabled
    • ​Install kubectl on Mac
    • Install k3d on Mac, on terminal use curl command
    • ​Homebrew (Homebrew is available for MacOS) using the below command to install curl
    brew install curl
    curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG=v4.4.8 bash

Create Lightweight Kubernetes Cluster

Ensure that the docker is running and you have the admin privileges for the device. Run the following in the terminal/command prompt.
Note: Make sure the k3d is already installed as part of the pre-requisites.
Start with the setup of the lightweight Kubernetes cluster on your local machine/VM. Execute all the instructions as admin/root/sudoer to give the installer full access to provision the required system resources/folders etc.
  1. 1.
    Create the "Kube" directory in the desired place (ensure you use the right dir path if it is different from the example) and change permission. This is used as k3d cluster persistent storage to store metadata and container logs.
    • For Linux/Mac
      1. 1.
        cd ~
        mkdir kube
        chmod 777 kube
        cd kube
        pwd #copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path
    • For Windows
      1. 1.
        cd D:\
        mkdir kube
        wsl chmod -R 777 kube
        cd kube
        pwd #copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path
  2. 2.
    Create a k3d cluster with a single master node and 2 agents (Worker Nodes) and mount the above-created directory (for data persistence). Here's where it stores all the metadata and persists the logs of the workloads.
    • k3d cluster create --k3s-server-arg "--no-deploy=traefik" --agents 2 -v "/home/<user_name>/kube:/kube@agent[0,1]" -v "/home/<user_name>/kube:/kube@server[0]" --port "80:80@loadbalancer"
      ​
    NOTE: Update "/home/<user_name>/kube" path in the above cmd with your respective absolute path.
  3. 3.
    Once the cluster creation is successful, get the kubeconfig file, that enables you to connect to the cluster.
    • k3d kubeconfig get k3s-default > myk3dconfig
      export KUBECONFIG=<path-to-your-kube_config>
    • kubectl config use-context k3d-k3s-default --kubeconfig=myk3dconfig
  4. 4.
    Verify the cluster creation by running the following commands from your local machine where the kubectl is installed. It provides the sample output as below if everything works fine.
    • kubectl cluster-info
      OutPut
      Kubernetes control plane is running at https://0.0.0.0:33931
      CoreDNS is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
      Metrics-server is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
      ​
    • k3d cluster list
      OutPut
      NAME SERVERS AGENTS LOADBALANCER
      k3s-default 1/1 2/2 true
      ​
    • kubectl get nodes
      OutPut
      NAME STATUS ROLES AGE VERSION
      k3d-k3s-default-agent-0 Ready <none> 3d18h v1.21.1+k3s1
      k3d-k3s-default-agent-1 Ready <none> 3d18h v1.21.1+k3s1
      k3d-k3s-default-server-0 Ready control-plane,master 3d18h v1.21.1+k3s1
      ​
    • kubectl top nodes
      OutPut
      W0625 07:56:24.588781 12810 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
      NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
      k3d-k3s-default-agent-0 547m 6% 1505Mi 9%
      k3d-k3s-default-agent-1 40m 0% 2175Mi 13%
      k3d-k3s-default-server-0 59m 0% 2286Mi 14%
      ​
On a Cloud (AWS)

Pre-reads

Know what is terraform: https://youtu.be/h970ZBgKINg​

Pre-requisites

  1. 1.
    Install terraform for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph. It also helps in destroying the cluster in one go. The desired terraform version is 0.14.10.
  • If you already have a different version of the terraform version running install tfswitch which would allow you to have multiple terraform versions in the same machine and toggle between the desired versions.
2. ​Setup your AWS account.
3. If you are not using the eGov AWS account, open 80, 22, and 6550 ports in a default security group.

1. Launching VM/Instance using Terraform

Step 1: Clone the following DIGIT Devops GitRepo. You may have to install git and then run git clone it to your machine.
git clone -b quickstart https://github.com/egovernments/DIGIT-DevOps
Step 2: After cloning the repo CD into the folder DIGIT-DevOps and type the "code" command. This opens the visual editor and the DIGIT-DevOps repo files.
cd DIGIT-DevOps/infra-as-code/terraform/quickstart-aws-ec2
code
Step 3: Generate ssh key
Step 4: The following main.tf will create ec2 instance.
provider "aws" {
region = "${var.region}"
}
module "ssh_key" {
source = "../modules/instance/aws-ec2"
key_name = "${var.key_name}"
public_key = "${var.public_key}"
}
​
resource "aws_instance" "digit-quickstart-vm" {
ami = "${var.ami_name_value}"
instance_type = "${var.instance_type}"
key_name = module.ssh_key.ssh_key_name
monitoring = false
associate_public_ip_address = true
availability_zone = "ap-south-1b"
​
tags = {
Name = "${var.tag}"
}
}
Step 5: Update variables.tf​
variable "region" {
default = "ap-south-1"
}
​
variable "ami_name_value" {
default = "ami-0bb9e2d19522c61d4"
}
​
variable "instance_type" {
default = "c5.2xlarge"
}
​
## The VM capacity may depend on the H/W requirement of the number of DIGIT modules that you may try for the POC
## Eg:
​
​
variable "tag" {
default = "digit-quickstart-vm" ## change tag name eg. digit-quickstart-vm_your-name_name
}
​
## change ssh public_key with your public ssh key
variable "public_key" {
default = "ssh-rsa <asfas>/Gy6w0PPSnnfl/AWXO7ckFtEXQbdz9Y15zeUFKgUsbklXxmC6D37BkPGu+IjCZSOttPV+PRM0Dnf0jQLvMV0UhEkguD9ALC5xikqNlFyPH5bGetWDxtLbn61tnoOIYG6lXAdk2Oe35yWWt3ZgcccWtYuRwDo0ofBwY9jWOkEcCefDyYg+S7h1VzNsbB9DsFv0vPcaxHcZK8bLdyhnz1+9rXy/flbiS5kE0O97aZ4zm4wAmqiivN2wWhUez18k2Mcs= demo@demo"
description = "ssh key"
}
​
​
variable "key_name" {
default = "digit-quickstart-vm" ## change ssh key_name eg.digit-quickstart-vm-your_name
description = "ssh key name"
}
Step 6: Export AWS Profile.
export AWS_PROFILE=digit-quickstart-poc
Step 7: Terraform Execution
cd DIGIT-DevOps/infra-as-code/terraform/quickstart-aws-ec2
terraform init
terraform plan
terraform apply
Step 8: The terraform apply command displays the results of creating the resource defined in the configuration files.
Note: Copy the "Public IP" from the outputs.
module.ssh_key.aws_key_pair.ssh_key: Creating...
module.ssh_key.aws_key_pair.ssh_key: Creation complete after 0s [id=digit-quickstart-vm]
aws_instance.digit-quickstart-poc: Creating...
aws_instance.digit-quickstart-poc: Still creating... [10s elapsed]
aws_instance.digit-quickstart-poc: Creation complete after 12s [id=i-05764e3b780d836dd]
​
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
​
Outputs:
​
public_ip = "<some_public_ip>"
Step 9: The terraform output provides the VM/instance public IP. Use the same to log in.
ssh -i <path_to_ssh_private_key> ubuntu@<your_vm_public_ip>
NOTE: replace IP <your_vm_public_ip> with your respective public IP
After logging into the newly provisioned AWS ec2 instance perform the steps below:

2. Create a Lightweight Kubernetes Cluster

Ensure that the VM meets the prescribed pre-requisites. Execute the commands below. In case, you are trying to set up on any other VM of your choice - ensure that the mentioned software/tools are installed and you have admin access to the machine.
To set up the lightweight Kubernetes cluster on your VM - execute all the instructions as admin/root/sudoer to give the installer full access to provision the required system resources/folders etc.
  • Create "Kube" directory in the desired place (ensure you use the right dir path if it is different from the example) and change permission accordingly. This will be used as k3d cluster persistent storage to store all the metadata and container logs.
sudo mkdir ~/kube
sudo chmod -R 777 ~/kube
cd ~/kube
pwd
​
#copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path
eg: /home/<your_user_name>/kube
  • Create a k3d cluster with a single master node and 2 agents (Worker Nodes) and mount the above-created directory (for data persistence). Here's where it stores all the metadata and persists the logs of the workloads.
Notes:
  • Update "/home/<your_user_name>/kube" path in the cmd below using the respective absolute path.
  • To route the traffic to the above-created cluster to connect to the cluster from the local machine/laptop using kubeconfig file.
  • Update <VM_Private_IP> with VM Private IP, use the below command on the VM terminal to get private IP.
ip a | grep "inet " | grep -v "127.0.0.1" | awk -F " " '{print $2}'|awk -F "/" '{print $1}'|head -n1
  • Update <VM_Public_IP> with VM's Public IP which you used to ssh into VM.
sudo k3d cluster create \
--api-port <VM_Private_IP>:6550 \
--k3s-server-arg "--no-deploy=traefik" \
--k3s-server-arg "--tls-san=<VM_Public_IP>" \
--agents 2 \
-v "/home/<your_user_name>/kube:/kube@agent[0,1]" \
-v "/home/<your_user_name>/kube:/kube@server[0]" \
--port "80:80@loadbalancer"
​
  • Once the cluster is created successfully, fetch the kubeconfig file that allows you to connect to the cluster.
  • sudo k3d kubeconfig get k3s-default > myk3dconfig
  • sudo kubectl config use-context k3d-k3s-default --kubeconfig=myk3dconfig
  • Verify the cluster creation by running the following commands from your AWS VM where the kubectl is installed. It gives the sample output as below if everything works fine.
  • sudo kubectl cluster-info
    OutPut
Kubernetes control plane is running at https://0.0.0.0:33931
CoreDNS is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
  • sudo k3d cluster list
    OutPut
    NAME SERVERS AGENTS LOADBALANCER
    k3s-default 1/1 2/2 true
    ​
  • sudo kubectl get nodes
    OutPut
    NAME STATUS ROLES AGE VERSION
    k3d-k3s-default-agent-0 Ready <none> 3d18h v1.21.1+k3s1
    k3d-k3s-default-agent-1 Ready <none> 3d18h v1.21.1+k3s1
    k3d-k3s-default-server-0 Ready control-plane,master 3d18h v1.21.1+k3s1
    ​
  • sudo kubectl top nodes
    OutPut
    W0625 07:56:24.588781 12810 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
    NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
    k3d-k3s-default-agent-0 547m 6% 1505Mi 9%
    k3d-k3s-default-agent-1 40m 0% 2175Mi 13%
    k3d-k3s-default-server-0 59m 0% 2286Mi 14%

To access the k3s cluster from local

  1. 1.
    Copy myk3dconfig to your local machine/laptop
  2. 2.
    Replace the clusters.cluster.server Private IP with Public Ip from your myk3dconfig.
For reference observe the below kubeconfig:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data:
server: https://<VM_Private_IP>:6550 //replace <VM_Private_IP> with <VM_Public_IP>
name: k3d-k3s-default
contexts:
- context:
cluster: k3d-k3s-default
user: admin@k3d-k3s-default
name: k3d-k3s-default
current-context: k3d-k3s-default
kind: Config
preferences: {}
users:
- name: admin@k3d-k3s-default
user:
client-certificate-data:
client-key-data:
​
3. Setup kubeconfig.
export KUBECONFIG=<path-to-your-kube_config>
4. To check the nodes locally, run the below command -
kubectl get nodes --kubeconfig=<path-to-your-kube_config>

Deploy DIGIT

Now you can start with the ​Deployment.

Destroy VM/Instance

Once you are done with the Quickstart setup, destroy the terraform resources.
cd DIGIT-DevOps/infra-as-code/terraform/digit-quickstart-aws
terraform destroy
If the above steps are completed successfully, your cluster is now up and running ready to proceed with the DIGIT deployment.
​
​
Previous
Quick Setup
Next
2. Deployment
Last modified 5mo ago
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.