DIGIT Core
PlatformDomainsAcademyDesign SystemFeedback
2.8
2.8
  • ☑️Introducing DIGIT Platform
    • DIGIT - Value Proposition
  • Platform
    • 🔎Overview
      • Principles
      • Architecture
        • Service Architecture
        • Infrastructure Architecture
        • Deployment Architecture
      • Technology
        • API Gateway
        • Open Source Tools
      • Checklists
        • API Checklist
        • Security Checklist
          • Security Guidelines Handbook
          • Security Flow - Exemplar
        • Performance Checklist
        • Deployment Checklist
      • UI Frameworks
        • React UI Framework
    • 🔧Core Services
      • Workflow Service
        • Setting Up Workflows
        • Configuring Workflows For An Entity
        • Workflow Auto Escalation
        • Migration To Workflow 2.0
      • Location Services
      • User Services
      • Access Control Services
      • PDF Generation Service
      • MDMS (Master Data Management Service)
        • Setting up Master Data
          • MDMS Overview
          • MDMS Rewritten
          • Configuring Tenants
          • Configuring Master Data
          • Adding New Master
          • State Level Vs City Level Master
      • Payment Gateway Service
      • User Session Management
      • Indexer Service
        • Indexer Configuration
      • URL Shortening Service
      • XState Core Chatbot
        • Xstate-Chatbot Message Localisation
        • XState-Chatbot Integration Document
      • NLP Engine Service
        • NLP Chatbot
      • SMS Template Approval Process
      • Telemetry Service
      • Document Uploader Service
      • Notification Enhancement Based On Different Channel
      • Report Service
        • Configuring New Reports
          • Impact Of Heavy Reports On Platform
          • Types Of Reports Used In Report Service
      • SMS Notification Service
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
      • Survey Service
      • Persister Service
        • Persister Configuration
      • Encryption Service
        • Encryption Client Library
        • User Data Security Architecture
        • Guidelines for supporting User Privacy in a module
      • FileStore Service
      • ID Generation Service
      • Localization Service
        • Configuring Localization
          • Setup Base Product Localization
          • Configure SMS and Email
      • Email Notification Service
      • Searcher Service
      • Zuul Service
      • User OTP Service
      • OTP Service
      • Chatbot Service
      • National Dashboard Ingest
        • National Dashboard API Performance Testing Specs and Benchmark
        • National Dashboard: Steps for Index Creation
        • National Dashboard Adaptor Service
          • Deployment of Airflow DAG
          • Trigger Airflow DAG
          • Configure Airflow
          • Insert & Delete Data - Steps
          • Important Links & Credentials
          • Code Structure
          • KT Sessions
          • Pre-requisites For Enabling Adaptor
        • Revenue Maximisation
      • Audit Service
        • Signed Audit Performance Testing Results
      • Service Request
      • Self Contained Service Architecture (HLD)
      • Accelerators
        • Inbox Service
    • ✏️API Specifications
      • User
      • Access Control
      • Employee
      • Location
      • Localisation
      • Encryption
      • Indexer
      • File Store
      • Collection
      • DSS Ingest
      • HRMS
      • National Dashboard Ingest
      • WhatsApp Chatbot
      • Master Data Management
      • ID Generation
      • URL Shortner
      • Workflow Service
      • Workflow v2
      • Document Uploader Service
      • OTP Service
      • Reporting Service
      • PDF Generation Service
      • Payment Gateway Service
    • 🔐Data Protection & Privacy
      • Data Protection & Privacy Definitions
      • Legal Obligations For Privacy - eGov
      • Data Protection & Privacy - Global Best Practices
      • Guidelines
        • Platform Owner Guidelines
        • Implementing Agencies Guidelines
        • Admin Guidelines
        • Program Owner Guidelines
        • Data Security and Data Privacy
      • Data Privacy Policy Templates
        • eGov Data Privacy Policy
        • Implementing Agency Privacy Policy
        • Admin & Program Owner Privacy Policy
        • Supporting Agency Privacy Policy
      • Global Standards For All Roles
    • ▶️Get Started
      • Install DIGIT
      • Access DIGIT
      • Sandbox
      • Training and Certification
        • Training Resources
    • ⚒️Integrations
      • Payment
      • Notification
      • Transaction
      • Verification
      • View
      • Calculation
    • 🛣️Roadmap
    • 🎬Open Events
    • 👩‍💻Source Code
    • 👁️Project Plan
    • 📋Discussion Board
    • 🤝Contribute
  • Guides
    • 📓Installation Guide
      • DIGIT Deployment
      • Quick Setup
        • DIGIT Installation on Azure
        • DIGIT Installation on AWS
      • Production Setup
        • AWS
          • 1. Pre-requisites
          • 2. Understanding EKS
          • 3. Setup AWS Account
          • 4. Provisioning Infra Using Terraform
          • 5. Prepare Deployment Config
          • 6. Deploy DIGIT
          • 7. Bootstrap DIGIT
          • 8. Productionize DIGIT
          • FAQ
        • Azure
          • 1. Azure Pre-requisites
          • 2. Understanding AKS
          • 3. Infra-as-code (Terraform)
        • SDC
          • 1. SDC Pre-requisites
          • 2. Infra-as-code (Kubespray)
          • CI/CD Setup On SDC
        • CI/CD Set Up
          • CI/CD Build Job Pipeline Setup
        • Prepare Helm Release Chart
        • Deployment - Key Concepts
          • Security Practices
          • Readiness & Liveness
          • Resource Requests & Limits
          • Deploying DIGIT Services
          • Deployment Architecture
          • Routing Traffic
          • Backbone Deployment
    • 💽Data Setup Guide
      • User Module
      • Localisation Module
      • Location Module
    • 🚥Design Guide
      • Model Requirements
      • Design Services
      • Design User Interface
      • Checklists
    • ⚒️Developer Guide
      • Pre-requisites Training Resources
      • Backend Developer Guide
        • Section 0: Prep
          • Development Pre-requisites
          • Design Inputs
            • High Level Design
            • Low Level Design
          • Development Environment Setup
        • Section 1: Create Project
          • Generate Project Using API Specs
          • Create Database
          • Configure Application Properties
          • Import Core Models
          • Implement Repository Layer
          • Create Validation & Enrichment Layers
          • Implement Service Layer
          • Build The Web Layer
        • Section 2: Integrate Persister & Kafka
          • Add Kafka Configuration
          • Implement Kafka Producer & Consumer
          • Add Persister Configuration
          • Enable Signed Audit
          • Run Application
        • Section 3: Integrate Microservices
          • Integrate IDGen Service
          • Integrate User Service
          • Add MDMS Configuration
          • Integrate MDMS Service
          • Add Workflow Configuration
          • Integrate Workflow Service
          • Integrate URL Shortener Service
        • Section 4: Integrate Billing & Payment
          • Custom Calculator Service
          • Integrate Calculator Service
          • Payment Back Update
        • Section 5: Other Advanced Integrations
          • Add Indexer Configuration
          • Certificate Generation
        • Section 6: Run Final Application
        • Section 7: Build & Deploy Instructions
        • FAQs
      • Flutter UI Developer Guide
        • Introduction to Flutter
          • Flutter - Key Features
          • Flutter Architecture & Approach
          • Flutter Pre-Requisites
        • Setup Development Environment
          • Flutter Installation & Setup Guide
          • Setup Device Emulators/Simulators
          • Run Application
        • Build User Interfaces
          • Create Form Screen
        • Build Deploy & Publish
          • Build & Deploy Flutter Web Application
          • Generate Android APKs & App Bundles
          • Publishing App Bundle To Play Store
        • State Management With Provider & Bloc
          • Provider State Management
          • BloC State Management
        • Best Practices & Tips
        • Troubleshooting
      • UI Developer Guide
        • DIGIT-UI
        • Android Web View & How To Generate APK
        • DIGIT UI Development Pre-requisites
        • UI Configuration (DevOps)
        • Local Development Setup
        • Run Application
        • Create New Screen In DIGIT-UI
          • Create Screen (FormComposer)
          • Inbox/Search Screen
          • Workflow Component
        • Customisation
          • Integrate External Web Application/UI With DIGIT UI
          • Utility - Pre-Process MDMS Configuration
          • CSS Customisation
        • Citizen Module Setup
          • Sample screenshots
          • Project Structure
          • Install Dependency
          • Import Required Components
          • Write Citizen Module Code
          • Citizen Landing Screen
        • Employee Module Setup
          • Write Employee Module Code
        • Build & Deploy
        • Setup Monitoring Tools
        • FAQs
          • Troubleshoot Using Browser Network Tab
          • Debug Android App Using Chrome Browser
    • 🔄Operations Guide
      • DIGIT - Infra Overview
      • Setup Central Instance Infra
      • Central Monitoring Dashboard Setup
      • Kubernetes
        • RBAC Management
        • DB Dump - Playground
      • Setup Jenkins - Docker way
      • GitOps
        • Git Client installation
        • GitHub organization creation
        • Adding new SSH key to it
        • GitHub repo creation
        • GitHub Team creation
        • Enabling Branch protection:
        • CODEOWNER Reviewers
        • Adding Users to the Git
        • Setting up an OAuth with GitHub
        • Fork (Fork the mdms,config repo with a tenant-specific branch)
      • Working with Kubernetes
        • Installation of Kubectl
      • Containerizing application using Docker
        • Creation of Dockerhub account
      • Infra provisioning using Terraform
        • Installation of Terraform
      • Customization of existing tf templates
      • Cert-Manager
        • Obtaining SSL certificates with the help of cluster-issuer
      • Moving Docker Images
      • Pre and post deployment checklist
      • Multi-tenancy Setup
      • Availability
        • Infrastructure
        • Backbone services
          • Database
          • Kafka
          • Kafka Connect
          • Elastic search
            • ElasticSearch Direct Upgrade
            • Elastic Search Rolling Upgrade
        • Core services
        • DIGIT apps
        • DSS dashboard
      • Observability
        • ES-Curator to clear old logs/indices
        • Monitoring
        • Tracing
        • Jaeger Tracing Setup
        • Logging
        • eGov Monitoring & Alerting Setup
        • eGov Logging Setup
      • Performance
        • What to monitor?
          • Infrastructure
          • Backbone services
          • Core services
        • Identifying bottlenecks
        • Solutions
      • Handling errors
      • Security
      • Reliability and disaster recovery
      • Privacy
      • Skillsets/hiring
      • Incident management processes
      • Kafka Troubleshooting Guide
        • How to clean up Kafka logs
        • How to change or reset consumer offset in Kafka?
      • SRE Rituals
      • FAQs
        • I am unable to login to the citizen or employee portal. The UI shows a spinner.
        • My DSS dashboard is not reflecting accurate numbers? What can I do?
      • Deployment using helm
        • Helm installation:
        • Helm chart creation
        • Helm chart customization
      • How to Dump Elasticsearch Indexes
      • Deploy Nginx-Ingress-Controller
      • Deployment Job Pipeline Setup
      • OAuth2-Proxy Setup
      • Jira Ticket Creation
  • Reference
    • 👉Setup Basics
      • Setup Requirements
        • Tech Enablement Training - Essential Skills and Pre-requisites
        • Tech Enablement Training (eDCR) - Essential Skills and Prerequisites
          • Development Control Rules (Digit-DCR)
          • eDCR Approach Guide
        • DIGIT Rollout Program Governance
        • DevOps Skills Requirements
        • Infra Requirements
        • Team Composition for DIGIT Implementation
        • Infra Best Practices
        • Operational Best Practices
        • Why Kubernetes For DIGIT
      • Supported Clouds
        • Google Cloud
        • Azure
        • AWS
        • VSphere
        • SDC
      • Deployment - Key Concepts
        • Security Practices
        • CI/CD
        • Readiness & Liveness
        • Resource Requests & Limits
      • Understanding ERP Stack
        • ERP Monolithic Architecture
        • ERP Hybrid Architecture
        • ERP Coexistence Architecture
        • APMDP-HYBRID-INFRA ARCHITECTURE
        • eGov SmartCity eGovernance Suite
        • ERP Deployment Process
        • ERP Release Process
        • ERP User Guide
      • Deploying DIGIT Services
        • Deployment Architecture
        • Routing Traffic
        • Backbone Deployment
      • Troubleshooting
        • Distributed Tracing
        • Logging
        • Monitoring & Alerts
    • 📥Reference Reads
      • Analytics
      • DevSecOps
      • Low Code No Code
        • Application Specification
      • Beneficiary Eligibility
      • Government and Open Digital Platforms
      • Microservices and Low Code No Code
      • Registries
      • Platform Orientation - Overview
    • 🔏Data Security
      • Signed Data Audit
      • Encryption Techniques
      • Approaches to handle Encrypted Data
    • ❕Privacy
    • 🕹️DevOps
      • 1. How DNS works
      • 2. Load Balancer
      • 3. SSL/Cert-manager
      • 4.Ingress,WAF
      • 5.VPC
      • 6.Subnets
      • 7.EKS
      • 8.Worker Node Group
      • 9.RDS
      • 10.NAT
      • 11.Internet Gateway
      • 12.Block Storage (EBS Volumes)
      • 13.Object Storage (S3)
      • 14. Telemetry
Powered by GitBook

All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page
  • Overview
  • Pre-requisites
  • Deployment Steps
  • Install Python
  • Disable Swap
  • Setup SSH using key-based authentication
  • Setup Ansible Controller Machine Setup Kubespray

Was this helpful?

  1. Guides
  2. Installation Guide
  3. Production Setup
  4. SDC

2. Infra-as-code (Kubespray)

Deploy DIGIT using Kubespray

Previous1. SDC Pre-requisitesNextCI/CD Setup On SDC

Last updated 1 year ago

Was this helpful?

Topics covered:

Overview

Kubespray is a composition of playbooks, , provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks. Kubespray provides:

  • a highly available cluster

  • composable attributes

  • support for most popular Linux distributions

  • continuous-integration tests

Pre-requisites

Before we can get started, we need a few prerequisites to be in place. This is what we are going to need:

  1. A host with Ansible installed. to learn more about Ansible. Find the Ansible installation details .

  2. You should also set up an SSH key pair to authenticate to the Kubernetes nodes without using a password. This permits Ansible to perform optimally.

  3. Few servers/hosts/VMs to serve as our targets to deploy Kubernetes. I am using Ubuntu 18.04, and my servers each have 4GB RAM and 2vCPUs. This is fine for my testing purposes, which I use to try out new things using Kubernetes. You need to be able to SSH into each of these nodes as root using the SSH key pair I mentioned above.

Deployment Steps

The above will do the following:

  • Create a new Linux User Account for use with Kubernetes on each node

  • Install Kubernetes and containers on each node

  • Configure the Master node

  • Join the Worker nodes to the new cluster

Install Python

Ansible needs Python to be installed on all the machines.

apt-get update && apt-get install python3-pip -y

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab

Setup SSH using key-based authentication

All the machines should be in the same network with Ubuntu or Centos installed.

ssh key should be generated from the Bastion machine and must be copied to all the servers part of your inventory.

  • Generate the ssh key ssh-keygen -t rsa

  • Copy over the public key to all nodes.

    ssh-copy-id root@<node-ip-address>

Setup Ansible Controller Machine Setup Kubespray

  • Clone the official repository

git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray
  • Install dependencies from requirements.txt

sudo pip install -r requirements.txt
  • Create Inventory

cp -rfp inventory/sample inventory/mycluster

where mycluster is the custom configuration name. Replace with whatever name you would like to assign to the current cluster.

Create inventory using an inventory generator.

declare -a IPS=(10.67.53.158 10.67.53.159 10.67.53.160 10.67.53.161 10.67.53.162 10.67.53.163 10.67.53.164)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Once it runs, you can see an inventory file that looks like the below:

all:
  hosts:
    node1:
      ansible_host: 10.67.53.158
      ip: 10.67.53.158
      access_ip: 10.67.53.158
    node2:
      ansible_host: 10.67.53.159
      ip: 10.67.53.159
      access_ip: 10.67.53.159
    node3:
      ansible_host: 10.67.53.160
      ip: 10.67.53.160
      access_ip: 10.67.53.160
    node4:
      ansible_host: 10.67.53.161
      ip: 10.67.53.161
      access_ip: 10.67.53.161
    node5:
      ansible_host: 10.67.53.162
      ip: 10.67.53.162
      access_ip: 10.67.53.162
    node6:
      ansible_host: 10.67.53.163
      ip: 10.67.53.163
      access_ip: 10.67.53.163
   node7:
      ansible_host: 10.67.53.164
      ip: 10.67.53.164
      access_ip: 10.67.53.164    
  children:
    kube-master:
      hosts:
        node1:
        node2:
        node3:
    kube-node:
      hosts:
        node1:
        node2:
        node3:
        node4:
        node5:
        node6:
        node7:
    etcd:
      hosts:
        node1:
        node2:
        node3:
        node4:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}
  • Review and change parameters under inventory/mycluster/group_vars

vim inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
## apiserver_loadbalancer_domain_name: "elb.some.domain"
apiserver_loadbalancer_domain_name: "10.211.55.101"
loadbalancer_apiserver:
  address: 10.211.55.101
  port: 443

Deploy Kubespray with Ansible Playbook - run the playbook as Ubuntu

  • The option --become is required for example writing SSL keys in /etc/, installing packages and interacting with various system daemons.

  • Note: Without --become - the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=ubuntu cluster.yml

Kubernetes cluster will be created with three masters and four nodes using the above process.

Kube config will be generated in a .Kubefolder. The cluster can be accessible via kubeconfig.

HA-Proxy

  • Install haproxy package in a haproxy machine that will be allocated for proxy

sudo apt-get install haproxy -y

  • IPs need to be whitelisted as per the requirements in the config.

sudo vim /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
       # ca-base /etc/ssl/certs
      #	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	# An alternative list with additional directives can be obtained from
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	#ssl-default-bind-ciphers 
	#ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
            timeout connect 5000
            timeout check   5000
            timeout client  30000
             timeout server  60000

frontend http-in
    bind *:80
    mode tcp
    default_backend http-servers
    http-request redirect scheme https unless { ssl_fc }

frontend https-in
    bind *:443
    mode tcp
    default_backend https-servers

frontend kube-in
    bind *:8383
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra 
    #acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    #tcp-request connection reject if !network_allowed
    default_backend kube-servers

frontend kube2-in
    bind *:6363
    mode tcp
    timeout client 3h
    #Jenkins_CD eGov_ACT eGov_Spectra
    acl network_allowed src 35.244.58.192 106.51.69.20 180.151.198.122 103.122.14.159 35.154.77.83 35.154.203.141 125.16.100.118 10.67.53.252  52.71.194.45 132.154.83.214 27.6.189.204 10.67.53.120
    tcp-request connection reject if !network_allowed
    default_backend kube-servers


backend http-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32080 send-proxy
        server srv5 10.67.53.162:32080 send-proxy
        server srv6 10.67.53.163:32080 send-proxy

backend https-servers
        mode tcp
        balance roundrobin
        server srv4 10.67.53.161:32443 send-proxy
        server srv5 10.67.53.162:32443 send-proxy
        server srv6 10.67.53.163:32443 send-proxy

backend kube-servers
        mode tcp     
        option log-health-checks
        timeout server 3h
        server master1 10.67.53.158:6443 check check-ssl verify none inter 10000
        server master2 10.67.53.159:6443 check check-ssl verify none inter 10000
        server master3 10.67.53.160:6443 check check-ssl verify none inter 10003
        balance roundrobin  

Volumes

Iscsi volumes will be provided by the SDC team as per the requisition and the same can be used for statefulsets.

sudo iscsiadm -m discovery -t sendtargets -p 10.67.49.8:3260

📓
Ansible
inventory
Click here
here
A brief introduction to Kubespray
Pre-requisites for deployment using Kubespray
Deployment steps