DIGIT Core
PlatformDomainsAcademyDesign SystemFeedback
2.9-LTS
2.9-LTS
  • 🖥️Platform
    • Overview
    • Why DIGIT?
    • Principles
    • Architecture
      • Service Architecture
      • Technology Architecture
        • Open Source Tools
      • Infrastructure Architecture
      • Deployment Architecture
    • API Specifications
      • Access Control
      • Boundary
      • Document Uploader
      • Encryption
      • File Store
      • ID Generation
      • Indexer
      • Localisation
      • Master Data Management
      • OTP
      • Payment Gateway
      • PDF Generation
      • URL Shortner
      • WhatsApp Chatbot
      • Workflow
    • Core Services
      • Access Control Services
      • Audit Service
        • Signed Audit Performance Testing Results
      • API Gateway
        • Configuring Gateway Rate Limiting
      • Boundary Service
        • Migrate Old Boundary Data - Steps
      • Email Notification Service
      • Encryption Service
        • Encryption Client Library
        • User Data Security Architecture
        • Guidelines for supporting User Privacy in a module
      • FileStore Service
      • ID Generation Service
      • Indexer Service
        • Indexer Configuration
      • Internal Gateway
      • Location
      • Localization Service
        • Configuring Localization
          • Setup Base Product Localisation
          • Configure SMS and Email
      • MDMS V2 (Master Data Management Service)
        • Adopt New MDMS - Steps
        • MDMS (Master Data Management Service)
          • Setting up Master Data
            • MDMS Overview
            • MDMS Rewritten
            • Configuring Tenants
            • Configuring Master Data
            • Adding New Master
            • State Level Vs City Level Master
        • MDMS Migration
      • OTP Service
      • Payment Gateway Service
      • PDF Generation Service
      • Persister Service
        • Persister Configuration
      • Service Request
      • SMS Notification Service
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
      • User
        • User Session Management
      • User OTP Service
      • URL Shortening Service
      • Workflow
        • Setting Up Workflows
        • Configuring Workflows For An Entity
        • Workflow Auto Escalation
        • Migration To Workflow 2.0
      • Libraries
        • Tracer Library
        • Encryption Client
      • Accelerators
        • Inbox Service
    • DIGIT: How-Tos
      • SMS Template Approval Process
      • Notification Enhancement Based On Different Channel
    • Releases
      • DIGIT 2.9 LTS
        • Test Automation
        • Release Checklist
        • Service Build Updates
          • Hotfix
        • Test Cases
        • Automated DIGIT Deployment
        • Upgrade Guide: Transitioning DIGIT Modules to Spring Boot Version 3.2.2
        • Postgres Upgrade: Service Code Changes
        • Updating RDS Version in AWS
        • LTS DIGIT Migration - v2.8 To v2.9
        • Changelog
        • Backup PostgreSQL Database In AWS - Steps
    • Source Code
  • 📓Guides
    • Installation Guide
      • Infrastructure Setup
        • AWS
          • 1. Pre-requisites
          • 2. Setup AWS Account
          • 3. Provision Infrastructure
          • FAQ
        • Azure
          • 1. Azure Pre-requisites
          • 2. Understanding AKS
          • 3. Infra-as-code (Terraform)
        • SDC
          • 1. SDC Pre-requisites
          • 2. Infra-as-code (Kubespray)
          • CI/CD Setup On SDC
        • CI/CD Set Up
          • CI/CD Build Job Pipeline Setup
      • DIGIT Deployment
        • Full Deployment
          • Deploy DIGIT
            • Prepare Deployment Configuration
        • Full Deployment (Beta)
          • Creating New HelmChart
          • Prepare Helm Release Chart
      • Quick Setup (AWS)
    • Data Setup Guide
      • Bootstrap DIGIT
      • Productionize DIGIT
      • User Module
      • Localisation Module
      • Location Module
      • MDMS - V2
    • Design Guide
      • Model Requirements
      • Design Services
      • Design User Interface
      • Checklists
    • Developer Guide
      • Pre-requisites Training Resources
      • Backend Developer Guide
        • Section 0: Prep
          • Development Pre-requisites
          • Design Inputs
            • High Level Design
            • Low Level Design
          • Development Environment Setup
        • Section 1: Create Project
          • Generate Project Using API Specs
          • Create Database
          • Configure Application Properties
          • Import Core Models
          • Implement Repository Layer
          • Create Validation & Enrichment Layers
          • Implement Service Layer
          • Build The Web Layer
        • Section 2: Integrate Persister & Kafka
          • Add Kafka Configuration
          • Implement Kafka Producer & Consumer
          • Add Persister Configuration
          • Enable Signed Audit
        • Section 3: Integrate Microservices
          • Integrate IDGen Service
          • Integrate User Service
          • Add MDMS Configuration
          • Integrate MDMS Service
          • Add Workflow Configuration
          • Integrate Workflow Service
          • Integrate URL Shortener Service
        • Section 4: Integrate Billing & Payment
          • Custom Calculator Service
          • Integrate Calculator Service
          • Payment Back Update
        • Section 5: Other Advanced Integrations
          • Add Indexer Configuration
          • Certificate Generation
        • Section 6: Run Final Application
        • Section 7: Build & Deploy Instructions
        • FAQs
      • UI Developer Guide
        • DIGIT-UI
          • UI Components Standardisation
            • DIGIT UI Core React Components
            • DIGIT UI Core Flutter Components
              • Input Field
              • Radio
              • Toggle
              • Button
              • Dropdown
              • Checkbox
              • Toast
              • Info Card
            • DIGIT UI Components v0.2.0
              • Foundation
                • Typography
                • Colour Pallete
                • Spacer
              • Atom
                • Accordion
                • Button
                • Checkbox
        • DIGIT UI Development Pre-requisites
        • UI Configuration (DevOps)
        • Local Development Setup
        • Run Application
        • Build & Deploy
        • Pre-defined Screens In DIGIT-UI
          • Create Screen (FormComposer)
          • Inbox/Search Screen
          • Workflow Component
        • Create a New UI Module/Package
          • Project Structure
          • Install Dependency
          • Module.js
          • Import Required Components
          • Common Hooks
        • Employee Module Setup
          • Write Employee Module Code
          • Create Form - Create Screen
        • Citizen Module Setup
          • Sample screenshots
          • Citizen Landing Screen
          • Write Citizen Module Code
        • Customisation
          • Integrate External Web Application/UI With DIGIT UI
          • Utility - Pre-Process MDMS Configuration
          • CSS Customisation
          • Kibana Dashboard Integration With DSS Module
          • Login Page
        • Setup Monitoring Tools
        • Android Web View & How To Generate APK
        • FAQs
          • Troubleshoot Using Browser Network Tab
          • Debug Android App Using Chrome Browser
      • Flutter (Mobile App) UI Developer Guide
        • Introduction to Flutter
          • Flutter - Key Features
          • Flutter Architecture & Approach
          • Flutter Pre-Requisites
        • Setup Development Environment
          • Flutter Installation & Setup Guide
          • Setup Device Emulators/Simulators
          • Run Application
        • Build User Interfaces
          • Create Form Screen
        • Build Deploy & Publish
          • Build & Deploy Flutter Web Application
          • Generate Android APKs & App Bundles
          • Publishing App Bundle To Play Store
        • State Management With Provider & Bloc
          • Provider State Management
          • BloC State Management
        • Best Practices & Tips
        • Troubleshooting
    • Operations Guide
      • DIGIT - Infra Overview
      • Kubernetes
        • RBAC Management
        • Database Dump - Playground
      • Setup Jenkins - Docker way
      • GitOps
        • Git Client installation
        • GitHub organization creation
        • Adding new SSH key to it
        • GitHub repo creation
        • GitHub Team creation
        • Enabling Branch protection:
        • CODEOWNER Reviewers
        • Adding Users to the Git
        • Setting up an OAuth with GitHub
        • Fork (Fork the mdms,config repo with a tenant-specific branch)
      • Working with Kubernetes
        • Installation of Kubectl
      • Containerizing application using Docker
        • Creation of Dockerhub account
      • Infra Provisioning Using Terraform
        • Installation of Terraform
      • Customise Existing Terraform Templates
      • Cert-Manager
        • Obtaining SSL certificates with the help of cluster-issuer
      • Moving Docker Images
      • Pre and post deployment checklist
      • Multi-tenancy Setup
      • Availability
        • Infrastructure
        • Backbone services
          • Database
          • Kafka
          • Kafka Connect
          • Elastic search
            • Elastic Search Rolling Upgrade
            • ElasticSearch Direct Upgrade
        • Core services
        • DIGIT apps
        • DSS dashboard
      • Observability
        • ES-Curator - Clear Old Logs/indices
        • Monitoring
        • Environment Changes
        • Tracing
        • Jaeger Tracing Setup
        • Logging
        • eGov Monitoring & Alerting Setup
        • eGov Logging Setup
      • Performance
        • What to monitor?
          • Infrastructure
          • Backbone services
          • Core services
        • Identifying bottlenecks
        • Solutions
      • Handling errors
      • Security
      • Reliability and disaster recovery
      • Privacy
      • Skillsets/hiring
      • Incident management processes
      • Kafka Troubleshooting Guide
        • How to clean up Kafka logs
        • How to change or reset consumer offset in Kafka?
      • SRE Rituals
      • FAQs
        • I am unable to login to the citizen or employee portal. The UI shows a spinner.
        • My DSS dashboard is not reflecting accurate numbers? What can I do?
      • Deployment using helm
        • Helm Installation
        • Helm chart creation
        • Helm chart customization
      • How to Dump Elasticsearch Indexes
      • Deploy Nginx-Ingress-Controller
      • Deployment Job Pipeline Setup
      • OAuth2-Proxy Setup
      • Jira Ticket Creation
    • Implementation Guide
    • Security & Privacy Guide
      • Security & Privacy Guidelines For Product Developers
      • Security & Privacy Guidelines For Solution Implementing Agencies
      • Security & Privacy Guidelines For Program Owners
  • 🚀Accelerators
    • UI Frameworks
      • Service Build Updates
    • Integrations
      • Payment
      • Notification
      • Transaction
      • Verification
      • View
      • Calculation
    • Concepts
      • Deployment - Key Concepts
        • Security Practices
        • Readiness & Liveness
        • Resource Requests & Limits
        • Deploying DIGIT Services
        • Deployment Architecture
        • Routing Traffic
        • Backbone Deployment
    • API Playground
    • Sandbox
    • Checklists
      • API Checklist
      • Security Checklist
        • Security Guidelines Handbook
        • Security Flow - Exemplar
      • Performance Checklist
      • Deployment Checklist
    • Contribute
    • Discussion Board
    • Academy
    • Events
Powered by GitBook

All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page

Was this helpful?

Last updated 2 months ago

Was this helpful?

Pre-reads

This doc covers the steps on how to deploy an OpenTelemetry collector on Kubernetes. We will then use an OTEL instrumented (Go) application provided by OpenTelemetry to send traces to the Collector. From there, we will bring the trace data to a Jaeger collector. Finally, the traces will be visualised using the Jaeger UI.

This image shows the flow between the application, OpenTelemetry collector and Jaeger.

Pre-requisites

To begin, we need a Kubernetes cluster you can use any of your existing Kubernetes clusters with the apx 2vCPUs, 4GB RAM, and 100GB Storage.

Local Kubernetes Cluster Setup

Skip this in case you have the existing cluster.

To use NodePort with Kind, we need to first enable it.

vim kind-config.yaml

Create the cluster with: kind create cluster --config kind-config.yaml

Once our Kubernetes cluster is up, we can start deploying Jaeger.

What is Jaeger?

Why do we need it?

  • Distributed tracing monitoring

  • Performance and latency optimisation

  • Root cause analysis

  • Service dependency analysis

Deploy Jaeger

Operators are pieces of software that ease the operational complexity of running another piece of software.

Deploy Jaeger Operator

You first install the Jaeger Operator on Kubernetes. This operator will then watch for new Jaeger custom resources (CR).

There are different ways of installing the Jaeger Operator on Kubernetes:

  • using Helm

  • using Deployment files

Since version 1.31 the Jaeger Operator uses webhooks to validate Jaeger custom resources (CRs). This requires an installed version of the cert-manager.

Installing Cert-Manager

Installation of cert-manager of is very simple, just run:

By default, cert-manager will be installed into the cert-manager namespace.

With cert-manager installed, let’s continue with the deployment of Jaeger

Installing Jaeger Operator Using Helm

Add the Jaeger Tracing Helm repository:

To install the chart with the release name my-release (in the default namespace)

You can also install a specific version of the helm chart:

Verify that it’s installed on Kubernetes:

helm list -A

You can also deploy the Jaeger operator using deployment files.

kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.36.0/jaeger-operator.yaml

At this point, there should be a jaeger-operator deployment available.

kubectl get deployment my-jaeger-operator

The operator is now ready to create Jaeger instances.

Deploy Jaeger All-in-One

The operator that we just installed doesn’t do anything itself, it just means that we can create jaeger resources/instances that we want the jaeger operator to manage.

The simplest possible way to create a Jaeger instance is by deploying the All-in-one strategy, which installs the all-in-one image, and includes the agents, collector, query and the Jaeger UI in a single pod using in-memory storage.

Create a yaml file like the following. The name of the Jaeger instance will be simplest

vim simplest.yaml

kubectl apply -f simplest.yaml

After a little while, a new in-memory all-in-one instance of Jaeger will be available, suitable for quick demos and development purposes.

When the Jaeger instance is up and running, we can check the pods and services.

kubectl get pods

kubectl get services

To get the pod name, query for the pods belonging to the simplest Jaeger instance:

Query the logs from the pod:

kubectl logs -l app.kubernetes.io/instance=simplest

Let’s open the Jaeger UI

kubectl port-forward svc/simplest-query 16686:16686

Jaeger UI

Deploy Open Telemetry Collector

vim otel-collector.yaml

Make sure to change the name of the jaeger collector (exporter) to match the one we deployed above. In our case, that would be:

Apply the file with: kubectl apply -f otel-collector.yaml

Verify that the OpenTelemetry Collector is up and running.

kubectl get deployment

kubectl logs deployment/otel-collector

Run Application

Time to send some trace data to our OpenTelemetry collector.

Remember, that the application access the Kubernetes cluster through a NodePort on port 30080. The Kubernetes service will bind the 4317 port used to access the OTLP receiver to port 30080 on the Kubernetes node.

go run main.go

Viewing the data

Let’s check out the telemetry data generated by our sample application

Again, we can use port-forwarding to access Jaeger UI.

Under Service select test-service to view the generated traces.

Done

This document has covered how we deploy an OpenTelemetry collector on Kubernetes. Then we sent trace data to this collector using an Otel SDK instrumented application written in Go. From there, the traces were sent to a Jaeger collector and visualised in Jaeger UI.

This OpenTelemetry provides a complete demo on how you can deploy OpenTelemetry on Kubernetes, we can use this as a starting point.

In case, you don't have the ready Kubernetes but you have a good local machine with at least 4GB RAM left, you can use a local instance of . The will access this Kubernetes cluster through a NodePort (on port 30080). So make sure this port is free.

can be used to port forward to the kind nodes. This is a cross-platform option to get traffic into your kind cluster.

is an open-source distributed tracing system for tracing transactions between distributed services. It’s used for monitoring and troubleshooting complex environments. By doing this, we can view traces and analyse the application’s behaviour.

Using a tracing system (like Jaeger) is especially important in environments since they are considered a lot more difficult to debug than a single monolithic application.

Problems that addresses?

To deploy Jaeger on the Kubernetes cluster, we can make use of the .

Before you start, pay attention to the section.

is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

You can verify the installation by following the instructions

Jump over to and search for jaeger-operator

helm repo add jaegertracing

Use port-forwarding to access the

To deploy the OpenTelemetry collector, we will use this file as a starting point. The yaml file consists of a ConfigMap, Service and a Deployment.

Also, pay attention to . This part creates the receiver on the Collector side and opens up the port 4317 for receiving traces, which enables the application to send data to the OpenTelemetry Collector.

By doing so, it makes it possible for us to access the Collector by using the static address <node-ip>:30080. In case you are running a local cluster, this will be localhost:30080.

This contains an (SDK) instrumented application written in Go, that simulates an application.

Open the web-browser and go to

The service name is specified in the main.go .

The will access this Kubernetes cluster through a NodePort (on port 30080). The URL is specified here:

All content on this page by is licensed under a .

conn, err := grpc.DialContext(ctx, "localhost:30080", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())
kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:- role: control-plane  # port forward 30080 on the host to 30080 on this node  extraPortMappings:  - containerPort: 30080    hostPort: 30080- role: worker
Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 ✓ Preparing nodes 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜Set kubectl context to "kind-kind"You can now use your cluster with:kubectl cluster-info --context kind-kindThanks for using kind! 😊
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml
helm install my-release jaegertracing/jaeger-operator
helm install my-release jaegertracing/jaeger-operator --version 2.25.0
NAME                 READY   UP-TO-DATE   AVAILABLE   AGEmy-jaeger-operator   1/1     1            1           2m58s
apiVersion: jaegertracing.io/v1kind: Jaegermetadata:  name: simplest
NAME                     READY STATUS    RESTARTS   AGEsimplest-656d7cf5c8-lff7b 1/1  Running   0          3m55s
{"level":"info","ts":1660155049.86027,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to READY","system":"grpc","grpc_log":true}{"level":"info","ts":1660155049.8612773,"caller":"grpc/builder.go:120","msg":"Agent collector connection state change","dialTarget":":14250","status":"READY"}{"level":"info","ts":1660155049.8617437,"caller":"app/server.go:241","msg":"Starting HTTP server","port":16686,"addr":":16686"}{"level":"info","ts":1660155049.8621716,"caller":"app/server.go:260","msg":"Starting GRPC server","port":16685,"addr":":16685"}
Forwarding from 127.0.0.1:16686 -> 16686Forwarding from [::1]:16686 -> 16686
exporters:      jaeger:        endpoint: "simplest-collector.default.svc.cluster.local:14250"
...  otel-collector-config: |    receivers:      otlp:        protocols:          grpc:            endpoint: "0.0.0.0:4317"...
configmap/otel-collector-conf createdservice/otel-collector createddeployment.apps/otel-collector created
"Everything is ready. Begin running and processing data."
2022/08/10 20:31:37 Waiting for connection...2022/08/10 20:31:37 Doing really hard work (1 / 10)2022/08/10 20:31:38 Doing really hard work (2 / 10)2022/08/10 20:31:39 Doing really hard work (3 / 10)2022/08/10 20:31:40 Doing really hard work (4 / 10)2022/08/10 20:31:41 Doing really hard work (5 / 10)2022/08/10 20:31:42 Doing really hard work (6 / 10)2022/08/10 20:31:43 Doing really hard work (7 / 10)2022/08/10 20:31:44 Doing really hard work (8 / 10)2022/08/10 20:31:45 Doing really hard work (9 / 10)2022/08/10 20:31:46 Doing really hard work (10 / 10)2022/08/10 20:31:47 Done!
kubectl port-forward svc/simplest-query 16686:16686
res, err := resource.New(ctx,  resource.WithAttributes(   // the service name used to display traces in backends   semconv.ServiceNameKey.String("test-service"),  ),
conn, err := grpc.DialContext(ctx, "localhost:30080", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock()) if err != nil {  return nil, fmt.Errorf("failed to create gRPC connection to collector: %w", err) }
  1. 📓Guides
  2. Operations Guide
  3. Observability

Tracing

PreviousEnvironment ChangesNextJaeger Tracing Setup
  • Pre-reads
  • Pre-requisites
  • Local Kubernetes Cluster Setup
  • What is Jaeger?
  • Why do we need it?
  • Problems that Jaeger addresses?
  • Deploy Jaeger
  • Deploy Jaeger Operator
  • Installing Cert-Manager
  • Installing Jaeger Operator Using Helm
  • Deploy Jaeger All-in-One
  • Let’s open the Jaeger UI
  • Deploy Open Telemetry Collector
  • Run Application
  • Viewing the data
repository
Kind
application
Extra port mappings
Jaeger
microservices
microservices
Jaeger
Jaeger operator
Prerequisite
cert-manager
here
Artifact Hub
https://jaegertracing.github.io/helm-charts
Jaeger UI
otel-collector.yaml
receivers
Source
repository
http://127.0.0.1:16686/
file
application
https://github.com/egovernments/core-docs/blob/2.9-LTS/get-started/operations-guide/observability/broken-reference/README.md
​
eGov Foundation
Creative Commons Attribution 4.0 International License