4. Provisioning Infra Using Terraform

There are multiple options available to deploy the solution to the cloud. Here we provide the steps to use terraform Infra-as-code.

Overview

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy DIGIT. The following picture shows the various key components. (EKS, Worker Nodes, PostGres DB, EBS Volumes, Load Balancer).

EKS Architecture for DIGIT Setup

Considering the above deployment architecture, the following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent Volumes)

• RDS (PostGres)

• VPCs (Private network)

• Users to access, deploy and read-only

Pre-requisites

Following are the pre-requisites and installation steps for setting up DIGIT on AWS:

1. Install Golang

   • For Linux: Follow the instructions here to install Golang on Linux.

   • For Windows: Download the installer using the link here and follow the installation instructions.

   • For Mac: Download the installer using the link here and follow the installation instructions.

2. Install Helm - DIGIT services are packaged with Helm Charts

3. Install kubectl - CLI to connect to the Kubernetes cluster on your machine

4. Install cURL - for making API calls

5. Install Visual Studio Code - for better code visualization/editing capabilities

6. Install Postman - to run digit bootstrap scripts

7. Install Terraform - to provide infrastructure on AWS

8. Install AWS CLI and IAM Authenticator

Prepare AWS IAM User

1. Create an IAM User in your AWS account.

2. Generate ACCESS_KEY and SECRET_KEY for the IAM user.

3. Assign Administrator Access to the IAM user for necessary permissions.

Once the prerequisites are installed you can begin setting up DIGIT and its services.

Network Infrastructure Setup

The terraform script once executed performs all of the below infrastructure setups.

Amazon EKS requires subnets must be in at least two different availability zones.

1. Create AWS VPC (Virtual Private Cloud).

2. Create two public and two private Subnets in different availability zones.

3. Create an Internet Gateway to provide Internet access for services within VPC.

4. Create NAT Gateway in public subnets. It is used in private subnets to allow services to connect to the internet.

5. Create Routing Tables and associate subnets with them. Add required routing rules.

6. Create Security Groups and associate subnets with them. Add required routing rules.

7. EKS cluster setup

The main.tf inside the sample-aws folder contains the detailed resource definitions that should be provisioned.

The sample-aws terraform script is provided as a helper/guide. An experienced DevOps can choose to modify or customize this as per the organization's infra needs.

To provision infrastructure and set up DIGIT, follow the steps below:

1. Clone the DIGIT-DevOps repository:

git clone https://github.com/egovernments/DIGIT-DevOps.git

2. Navigate to the cloned repository and checkout the release-1.28-kubernetes branch:

cd DIGIT-DevOps 
git checkout release-1.28-kubernetes

3. Check if the correct credentials are configured using the command below. Refer to the attached doc to setup AWS Account on the local machine.

aws configure list

Make sure that the above command reflects the set AWS credentials. Proceed once the details are confirmed. (Refer to the AWS document in case of any doubts on how to set the credentials: Configuring the AWS CLI - AWS Command Line Interface )

4. Choose either method below to generate ssh key pairs

   • a. Use online website (not recommended in a production setup. To be only used for demo setups): https://8gwifi.org/sshfunctions.jsp

   • b. Use openssl:

openssl genpkey -algorithm RSA -out private_key.pem openssl rsa -pubout -in private_key.pem -out public_key.pem

5. Add the public key to your GitHub account.

6. Open input.yaml file in vscode. Use the below code to open it in VS code:

   code infra-as-code/terraform/sample-aws/input.yaml

   If the command does not work, open the file in VS code manually. Once the file is open, fill in the inputs. (If you are not using vscode, open it in any editor of your choice).

7. Fill in the inputs as per the regex mentioned in the comments.

8. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

cd infra-as-code/terraform/sample-aws 
go run ../scripts/init.go

Terraform Execution: Infrastructure Resources Provisioning

Once we are complete declaring the resources, begin with deploying all resources.

Run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. First CD into the following directory and run the following command to create the remote state.

### Create the remote-state first, remember that the state name should be unique

cd remote-state

terraform init

terraform plan

terraform apply

Once the remote state is created, it is time to provision DIGIT infra. Run the following commands:

### Once the remote state is created, you can create the DIGIT Infra

cd ..

terraform init

terraform plan

terraform apply

Important:

• DB password is asked for in the application stage. Remember the password you have provided. It should be at least 8 characters long. Otherwise, RDS provisioning will fail.

• The output of the apply command will be displayed on the console. Store this in a file somewhere. Values from this file will be used in the next step of deployment.

2. Use this link to get the kubeconfig from EKS for the cluster. The region code is the default region provided in the availability zones in variables.tf. For example - ap-south-1. EKS cluster name also should've been filled in variables.tf.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

3. Verify that you can connect to the cluster by running the following command

kubectl config use-context <cluster_name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-2.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-3.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2   
ip-192-168-xx-4.ap-south-1.compute.internal   Ready  45d   v1.15.10-eks-bac369   Amazon Linux 2 

At this point, your basic infra has been provisioned.

Note: Refer to the DIGIT deployment documentation to deploy DIGIT services.

Destroying Infra

To destroy the previously created infrastructure with Terraform, run the command below:

1. ELB is not deployed via Terraform. ELB was created at deployment time by the setup of Kubernetes Ingress. This has to be deleted manually by deleting the ingress service.

   • kubectl delete deployment nginx-ingress-controller -n <namespace>

   • kubectl delete svc nginx-ingress-controller -n <namespace>

     Note: Namespace can be either egov or jenkins.

2. Delete S3 buckets manually from the AWS console and verify if ELB got deleted.

   • In case of if ELB is not deleted, you need to delete ELB from the AWS console.

3. Run terraform destroy.

Sometimes all artefacts associated with a deployment cannot be deleted through Terraform. For example, RDS instances might have to be deleted manually. It is recommended to log in to the AWS management console and look through the infra to delete any remnants.​