DIGIT Core
PlatformDomainsAcademyDesign SystemFeedback
2.8
2.8
  • ☑️Introducing DIGIT Platform
    • DIGIT - Value Proposition
  • Platform
    • 🔎Overview
      • Principles
      • Architecture
        • Service Architecture
        • Infrastructure Architecture
        • Deployment Architecture
      • Technology
        • API Gateway
        • Open Source Tools
      • Checklists
        • API Checklist
        • Security Checklist
          • Security Guidelines Handbook
          • Security Flow - Exemplar
        • Performance Checklist
        • Deployment Checklist
      • UI Frameworks
        • React UI Framework
    • 🔧Core Services
      • Workflow Service
        • Setting Up Workflows
        • Configuring Workflows For An Entity
        • Workflow Auto Escalation
        • Migration To Workflow 2.0
      • Location Services
      • User Services
      • Access Control Services
      • PDF Generation Service
      • MDMS (Master Data Management Service)
        • Setting up Master Data
          • MDMS Overview
          • MDMS Rewritten
          • Configuring Tenants
          • Configuring Master Data
          • Adding New Master
          • State Level Vs City Level Master
      • Payment Gateway Service
      • User Session Management
      • Indexer Service
        • Indexer Configuration
      • URL Shortening Service
      • XState Core Chatbot
        • Xstate-Chatbot Message Localisation
        • XState-Chatbot Integration Document
      • NLP Engine Service
        • NLP Chatbot
      • SMS Template Approval Process
      • Telemetry Service
      • Document Uploader Service
      • Notification Enhancement Based On Different Channel
      • Report Service
        • Configuring New Reports
          • Impact Of Heavy Reports On Platform
          • Types Of Reports Used In Report Service
      • SMS Notification Service
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
      • Survey Service
      • Persister Service
        • Persister Configuration
      • Encryption Service
        • Encryption Client Library
        • User Data Security Architecture
        • Guidelines for supporting User Privacy in a module
      • FileStore Service
      • ID Generation Service
      • Localization Service
        • Configuring Localization
          • Setup Base Product Localization
          • Configure SMS and Email
      • Email Notification Service
      • Searcher Service
      • Zuul Service
      • User OTP Service
      • OTP Service
      • Chatbot Service
      • National Dashboard Ingest
        • National Dashboard API Performance Testing Specs and Benchmark
        • National Dashboard: Steps for Index Creation
        • National Dashboard Adaptor Service
          • Deployment of Airflow DAG
          • Trigger Airflow DAG
          • Configure Airflow
          • Insert & Delete Data - Steps
          • Important Links & Credentials
          • Code Structure
          • KT Sessions
          • Pre-requisites For Enabling Adaptor
        • Revenue Maximisation
      • Audit Service
        • Signed Audit Performance Testing Results
      • Service Request
      • Self Contained Service Architecture (HLD)
      • Accelerators
        • Inbox Service
    • ✏️API Specifications
      • User
      • Access Control
      • Employee
      • Location
      • Localisation
      • Encryption
      • Indexer
      • File Store
      • Collection
      • DSS Ingest
      • HRMS
      • National Dashboard Ingest
      • WhatsApp Chatbot
      • Master Data Management
      • ID Generation
      • URL Shortner
      • Workflow Service
      • Workflow v2
      • Document Uploader Service
      • OTP Service
      • Reporting Service
      • PDF Generation Service
      • Payment Gateway Service
    • 🔐Data Protection & Privacy
      • Data Protection & Privacy Definitions
      • Legal Obligations For Privacy - eGov
      • Data Protection & Privacy - Global Best Practices
      • Guidelines
        • Platform Owner Guidelines
        • Implementing Agencies Guidelines
        • Admin Guidelines
        • Program Owner Guidelines
        • Data Security and Data Privacy
      • Data Privacy Policy Templates
        • eGov Data Privacy Policy
        • Implementing Agency Privacy Policy
        • Admin & Program Owner Privacy Policy
        • Supporting Agency Privacy Policy
      • Global Standards For All Roles
    • ▶️Get Started
      • Install DIGIT
      • Access DIGIT
      • Sandbox
      • Training and Certification
        • Training Resources
    • ⚒️Integrations
      • Payment
      • Notification
      • Transaction
      • Verification
      • View
      • Calculation
    • 🛣️Roadmap
    • 🎬Open Events
    • 👩‍💻Source Code
    • 👁️Project Plan
    • 📋Discussion Board
    • 🤝Contribute
  • Guides
    • 📓Installation Guide
      • DIGIT Deployment
      • Quick Setup
        • DIGIT Installation on Azure
        • DIGIT Installation on AWS
      • Production Setup
        • AWS
          • 1. Pre-requisites
          • 2. Understanding EKS
          • 3. Setup AWS Account
          • 4. Provisioning Infra Using Terraform
          • 5. Prepare Deployment Config
          • 6. Deploy DIGIT
          • 7. Bootstrap DIGIT
          • 8. Productionize DIGIT
          • FAQ
        • Azure
          • 1. Azure Pre-requisites
          • 2. Understanding AKS
          • 3. Infra-as-code (Terraform)
        • SDC
          • 1. SDC Pre-requisites
          • 2. Infra-as-code (Kubespray)
          • CI/CD Setup On SDC
        • CI/CD Set Up
          • CI/CD Build Job Pipeline Setup
        • Prepare Helm Release Chart
        • Deployment - Key Concepts
          • Security Practices
          • Readiness & Liveness
          • Resource Requests & Limits
          • Deploying DIGIT Services
          • Deployment Architecture
          • Routing Traffic
          • Backbone Deployment
    • 💽Data Setup Guide
      • User Module
      • Localisation Module
      • Location Module
    • 🚥Design Guide
      • Model Requirements
      • Design Services
      • Design User Interface
      • Checklists
    • ⚒️Developer Guide
      • Pre-requisites Training Resources
      • Backend Developer Guide
        • Section 0: Prep
          • Development Pre-requisites
          • Design Inputs
            • High Level Design
            • Low Level Design
          • Development Environment Setup
        • Section 1: Create Project
          • Generate Project Using API Specs
          • Create Database
          • Configure Application Properties
          • Import Core Models
          • Implement Repository Layer
          • Create Validation & Enrichment Layers
          • Implement Service Layer
          • Build The Web Layer
        • Section 2: Integrate Persister & Kafka
          • Add Kafka Configuration
          • Implement Kafka Producer & Consumer
          • Add Persister Configuration
          • Enable Signed Audit
          • Run Application
        • Section 3: Integrate Microservices
          • Integrate IDGen Service
          • Integrate User Service
          • Add MDMS Configuration
          • Integrate MDMS Service
          • Add Workflow Configuration
          • Integrate Workflow Service
          • Integrate URL Shortener Service
        • Section 4: Integrate Billing & Payment
          • Custom Calculator Service
          • Integrate Calculator Service
          • Payment Back Update
        • Section 5: Other Advanced Integrations
          • Add Indexer Configuration
          • Certificate Generation
        • Section 6: Run Final Application
        • Section 7: Build & Deploy Instructions
        • FAQs
      • Flutter UI Developer Guide
        • Introduction to Flutter
          • Flutter - Key Features
          • Flutter Architecture & Approach
          • Flutter Pre-Requisites
        • Setup Development Environment
          • Flutter Installation & Setup Guide
          • Setup Device Emulators/Simulators
          • Run Application
        • Build User Interfaces
          • Create Form Screen
        • Build Deploy & Publish
          • Build & Deploy Flutter Web Application
          • Generate Android APKs & App Bundles
          • Publishing App Bundle To Play Store
        • State Management With Provider & Bloc
          • Provider State Management
          • BloC State Management
        • Best Practices & Tips
        • Troubleshooting
      • UI Developer Guide
        • DIGIT-UI
        • Android Web View & How To Generate APK
        • DIGIT UI Development Pre-requisites
        • UI Configuration (DevOps)
        • Local Development Setup
        • Run Application
        • Create New Screen In DIGIT-UI
          • Create Screen (FormComposer)
          • Inbox/Search Screen
          • Workflow Component
        • Customisation
          • Integrate External Web Application/UI With DIGIT UI
          • Utility - Pre-Process MDMS Configuration
          • CSS Customisation
        • Citizen Module Setup
          • Sample screenshots
          • Project Structure
          • Install Dependency
          • Import Required Components
          • Write Citizen Module Code
          • Citizen Landing Screen
        • Employee Module Setup
          • Write Employee Module Code
        • Build & Deploy
        • Setup Monitoring Tools
        • FAQs
          • Troubleshoot Using Browser Network Tab
          • Debug Android App Using Chrome Browser
    • 🔄Operations Guide
      • DIGIT - Infra Overview
      • Setup Central Instance Infra
      • Central Monitoring Dashboard Setup
      • Kubernetes
        • RBAC Management
        • DB Dump - Playground
      • Setup Jenkins - Docker way
      • GitOps
        • Git Client installation
        • GitHub organization creation
        • Adding new SSH key to it
        • GitHub repo creation
        • GitHub Team creation
        • Enabling Branch protection:
        • CODEOWNER Reviewers
        • Adding Users to the Git
        • Setting up an OAuth with GitHub
        • Fork (Fork the mdms,config repo with a tenant-specific branch)
      • Working with Kubernetes
        • Installation of Kubectl
      • Containerizing application using Docker
        • Creation of Dockerhub account
      • Infra provisioning using Terraform
        • Installation of Terraform
      • Customization of existing tf templates
      • Cert-Manager
        • Obtaining SSL certificates with the help of cluster-issuer
      • Moving Docker Images
      • Pre and post deployment checklist
      • Multi-tenancy Setup
      • Availability
        • Infrastructure
        • Backbone services
          • Database
          • Kafka
          • Kafka Connect
          • Elastic search
            • ElasticSearch Direct Upgrade
            • Elastic Search Rolling Upgrade
        • Core services
        • DIGIT apps
        • DSS dashboard
      • Observability
        • ES-Curator to clear old logs/indices
        • Monitoring
        • Tracing
        • Jaeger Tracing Setup
        • Logging
        • eGov Monitoring & Alerting Setup
        • eGov Logging Setup
      • Performance
        • What to monitor?
          • Infrastructure
          • Backbone services
          • Core services
        • Identifying bottlenecks
        • Solutions
      • Handling errors
      • Security
      • Reliability and disaster recovery
      • Privacy
      • Skillsets/hiring
      • Incident management processes
      • Kafka Troubleshooting Guide
        • How to clean up Kafka logs
        • How to change or reset consumer offset in Kafka?
      • SRE Rituals
      • FAQs
        • I am unable to login to the citizen or employee portal. The UI shows a spinner.
        • My DSS dashboard is not reflecting accurate numbers? What can I do?
      • Deployment using helm
        • Helm installation:
        • Helm chart creation
        • Helm chart customization
      • How to Dump Elasticsearch Indexes
      • Deploy Nginx-Ingress-Controller
      • Deployment Job Pipeline Setup
      • OAuth2-Proxy Setup
      • Jira Ticket Creation
  • Reference
    • 👉Setup Basics
      • Setup Requirements
        • Tech Enablement Training - Essential Skills and Pre-requisites
        • Tech Enablement Training (eDCR) - Essential Skills and Prerequisites
          • Development Control Rules (Digit-DCR)
          • eDCR Approach Guide
        • DIGIT Rollout Program Governance
        • DevOps Skills Requirements
        • Infra Requirements
        • Team Composition for DIGIT Implementation
        • Infra Best Practices
        • Operational Best Practices
        • Why Kubernetes For DIGIT
      • Supported Clouds
        • Google Cloud
        • Azure
        • AWS
        • VSphere
        • SDC
      • Deployment - Key Concepts
        • Security Practices
        • CI/CD
        • Readiness & Liveness
        • Resource Requests & Limits
      • Understanding ERP Stack
        • ERP Monolithic Architecture
        • ERP Hybrid Architecture
        • ERP Coexistence Architecture
        • APMDP-HYBRID-INFRA ARCHITECTURE
        • eGov SmartCity eGovernance Suite
        • ERP Deployment Process
        • ERP Release Process
        • ERP User Guide
      • Deploying DIGIT Services
        • Deployment Architecture
        • Routing Traffic
        • Backbone Deployment
      • Troubleshooting
        • Distributed Tracing
        • Logging
        • Monitoring & Alerts
    • 📥Reference Reads
      • Analytics
      • DevSecOps
      • Low Code No Code
        • Application Specification
      • Beneficiary Eligibility
      • Government and Open Digital Platforms
      • Microservices and Low Code No Code
      • Registries
      • Platform Orientation - Overview
    • 🔏Data Security
      • Signed Data Audit
      • Encryption Techniques
      • Approaches to handle Encrypted Data
    • ❕Privacy
    • 🕹️DevOps
      • 1. How DNS works
      • 2. Load Balancer
      • 3. SSL/Cert-manager
      • 4.Ingress,WAF
      • 5.VPC
      • 6.Subnets
      • 7.EKS
      • 8.Worker Node Group
      • 9.RDS
      • 10.NAT
      • 11.Internet Gateway
      • 12.Block Storage (EBS Volumes)
      • 13.Object Storage (S3)
      • 14. Telemetry
Powered by GitBook

All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page
  • Introduction
  • Architecture Components
  • The Goal
  • The Components & Why
  • nginx
  • Zuul
  • Spring Boot
  • Kafka
  • ElasticSearch
  • PostgreSQL
  • Docker
  • Kubernetes
  • Putting it All Together

Was this helpful?

  1. Reference
  2. Reference Reads

Platform Orientation - Overview

Introduction

This document serves as a briefing and overview of the core architecture and components of the platform for a new or unfamiliar developer. It seeks to address the what, why, and how of the platform at the time of writing. It is also meant to be a collaborative exercise, written by newbies for newbies, with future developers adding their own insights and learnings to this resource to have it grow with the platform over time.

This is NOT a technical reference or documentation. It is intended for orientation and will be written in natural language wherever possible. It is also limited in its scope to the general architecture of the back end, with little regard to how the systems necessarily converge to provide product solutions.

Architecture Components

The Goal

By the end of this document, you will be able to completely comprehend the following paragraph. It will equip you to understand the terminology, the tools, the features, and the implicit assumptions therein as well as provide you with solid grounded reasoning on why the architecture is the way that it is. The paragraph is an elevator pitch of the platform architecture, and it looks something like this:

In brief, the platform stack uses nginx servers with Zuul gateways to host Spring Boot microservices stored in Docker containers managed using Kubernetes. The servers rely on Kafka data streams to provide them with data that is indexed in ElasticSearch, and persisted in PostgreSQL databases.

Here’s what you need to know.

The Components & Why

nginx

Definition: nginx (pronounced “Engine X”) is a web server designed to serve dynamic HTTP content fast. It serves 32% of all active websites on the internet as of 2019, making it the world’s most popular web server.

A server in this context is a computer on a network that holds some form of content and provides it when needed i.e. “serves” it.

Functionality: Nginx uses a modular event-driven architecture to handle requests asynchronously, rather than through threads. “Event-driven” means it performs actions as a reaction to things happening in its environment (such as requests for information, or changes in values), as opposed to constantly staying in action to function (which is what threading does).

Why nginx: Nginx is substantially faster than Apache at a fraction of the processor cost because the narrow scope of a microservice server means that the configuration is highly specialized making it more efficient than a feature-rich server which would do more but run slower.

Because the platform microservices are all HTTP driven, a server that is optimized for fast dynamic HTTP processing makes logical sense.

Zuul

Definition: Zuul is an open-source API gateway service developed and provided by Netflix. An API Gateway is a service that manages access control to a server that is hosting an API, which means that it can handle things like service requests that involve sending and receiving program operation-specific data and parameters and is custom-built for that purpose.

Functionality: Zuul acts as a proxy, accepting all incoming API requests and authenticating them before delegating them to the microservice in question. This means that whenever an app or a product is requesting or calling a microservice, it is actually connecting to Zuul first, rather than directly to the server. Once Zuul okays the request, it hands off to the server.

Why Zuul: Zuul provides two benefits: it acts as a wrapper on the internal mechanics of the microservices, meaning that any internal functionality concerns are irrelevant to any external clients. It also simplifies the server gateway and access system, allowing for a single configuration of authentication protocols to suffice for every deployed microservice. In the absence of a common gateway, authentication would have to be individually defined on every server access point, which would be tedious and redundant.

Spring Boot

Definition: Spring is an application framework for Java, or the environment in which a java application runs. Spring Boot is an opinionated instance of the Spring framework, which means that it is automatically preconfigured in the way most Java application frameworks tend to be configured on average.

Functionality: The opinionated configuration of Spring Boot means that a developer does not need to be spending time and resources to install the libraries and dependencies required for a specific Java application. They are all present at the time of deployment and only highly specialized dependencies need to be installed after the fact.

Why Spring: Because the platform consists of a large number of microservices where the individual functionalities of a given service are defined very simply, it is unlikely that highly specialized dependencies will be required for a non-opinionated configuration to be required or viable. Therefore, an opinionated instance that includes all the commonly required dependencies by design is an ideal match for the framework requirements for a project such as this.

Kafka

Definition: Kafka is a real-time data streaming service. It allows other systems to subscribe or publish to a data stream (a sequence of data that updates asynchronously in real-time).

Functionality: Kafka acts as the backbone of the server architecture, handling data transfer between the databases and the microservices, as well as other platform entities that require access to data and functionality elements. It creates streams of information that services and network entities can either publish or subscribe to.

Why Kafka (or why Data Streaming): Data streaming in general, and Kafka in particular, address an important aspect of microservice architecture design. Inter-service communication plays a larger role in the functionality of such architecture over traditional service architectures, and being able to reliably and efficiently provide data to all the microservices active at a given time during runtime is essential to the platform working as intended.

With streaming, services that need data can request it independent of each other without affecting the functionality of others (a key advantage of a pub/sub model) and the data can be reliably expected to be up-to-date. With distributed streaming infrastructures like Kafka, scaling up to accommodate larger and more complex microservice deployments also becomes easier.

ElasticSearch

Definition: ElasticSearch is a search engine. It provides text-based search functionality across an indexed database.

Functionality: ElasticSearch allows for searching across all kinds of documents, including specifically schema-less JSON objects. It is quasi-real time, allows its database indices to be sharded (horizontally partitioned) with shard-level replication as well as distributed computation and storage.

ElasticSearch is complemented by Logstash, a data collection and logging system and Kibana, an analytics and visualization dashboard. These three tools, combined with Beats, a lightweight data shipper (not being used in the architecture) are collectively named the Elastic Stack.

Why the Elastic Stack: The Elastic stack is self-contained and highly functional, ideal for the “just works” configuration that is needed for scalable systems. Specifically, ElasticSearch is a more efficient method of searching the database since the query runtime is faster on indexed Elastic than on indexed relational databases. It works in tandem with the slower but more robust relational database to provide faster data access.

PostgreSQL

Definition: PostgreSQL is an open-source relational database management system developed by the Ingres team at the University of California, Berkeley.

Functionality: PostgreSQL is a fully functional RDBMS that is market competitive with other open source and proprietary database management tools. A full list of the features it offers would be slightly redundant to add to this document, but it could be introduced at a later date.

Why PostgreSQL: PostgreSQL has one real advantage over other forms of open source RDBMS in that it is slightly faster. MySQL will run slower on average on certain specific query cases and corner cases. Furthermore, there is a consensus in the platform development community that a move to PostgreSQL is inevitable in all but the most legacy of systems. Non-Postgres systems at large scale are only really being maintained because migration would be too resource-intensive to be worthwhile.

Docker

Definition: Docker is virtualization software that creates lightweight virtual environments called containers in which programs can be run with their own unique configuration of libraries, dependencies, and setups. Because all Docker containers run on one OS kernel, they are less resource intensive than virtual machines (which instantiate a new OS for every virtualization).

Functionality: Docker uses Linux functionality like cgroups (which allows for compartmentalizing hardware resources) and namespaces to isolate the containers without having to create a new instance of the kernel for every virtualization.

Docker containers are also ephemeral, in that they only exist for as long as it is needed for the app or service running within the container to perform the necessary task, after which it is cleaned up.

Why Docker: Virtualization and containers are advantageous for a distributed scaled system because of the ease of configuration for individual microservice functionality. This in turn lowers the size of the resultant code base, as well as allows for constant delivery (since the entire stack does not need to be taken down to instantiate a new container for a new microservice).

Kubernetes

Definition: Kubernetes is an open-source container orchestration platform that allows for automating the container deployment, maintenance, and scaling process.

Functionality: Kubernetes consolidates containers into pods, which are groups of containers guaranteed to be hosted in a single location and can share resources. These pods are then organized into services, where the containers are all intended to interact with each other. These are deployed in Kubernetes Nodes on the API server architecture, which are accessed by clients via the Kube-proxy interface.

Why Kubernetes: By design, Kubernetes and by extension the container architecture it facilitates, meet a lot of the concerns and requirements of microservice architectures. Over time, as the system complexity increases, the automation of container management means that the service can be scaled and managed without hindering functionality, provided the core design is consistent with the problem it is attempting to solve.

Putting it All Together

Revisiting the Elevator Pitch

In brief, the platform stack uses nginx servers with Zuul gateways to host Spring Boot microservices stored in Docker containers managed using Kubernetes. The servers rely on Kafka data streams to provide them with data that is stored indexed in ElasticSearch, and persisted in PostgreSQL databases.

Now that you have read the document, you should be better equipped to understand what that means, as well as the raison d’être for its current state. You should also be cognizant of the context in which the platform functions, and the nature of the solutions it is capable of providing.

Most importantly, you are now ready to jump into the technical documentation and be able to put it in perspective with the system at large, while being able to focus on the specific aspect with which you are concerned.

What Now?

Still doesn’t make sense? Feels like something is missing. Is everything in this document wrong and bad and you can’t believe someone actually wrote this stuff out? Don’t worry! This is a collaborative effort, and your contribution will be most welcome. Ping the author(s), leave a comment, or better yet, edit the document yourself and keep improving it. The more the better.

Over time, this document is intended to help any new team members become familiar and capable with the platform and anything you design worthy of adding to their knowledge should be added here.

If you’re good to go, however, then get in touch with your team and they will let you know what is next.

PreviousRegistriesNextData Security

Last updated 1 year ago

Was this helpful?

📥