DIGIT Urban
PlatformDomainsAcademyDesign SystemFeedback
v2.4
v2.4
  • DIGIT
  • Open Events
  • Architecture
  • Roadmap
  • Release Notes
    • MDMS Configuration & Service Build Updates
    • WhatsApp Channel Release Notes
    • eChallan Release Notes
    • Property Tax Release Notes
    • QA Automation Release Notes
    • Fire NOC Release Notes
  • Product Docs
    • Products & Modules
      • Trade License (TL)
        • TL Brochure
        • TL Roadmap
        • TL Module Functional Specifications
        • TL Workflows
        • TL Implementation Guide
        • TL Master Data Templates
          • Trade Type
          • Trade Sub Type
          • Trade Category
          • Trade License Fee
          • Structure Sub Type
          • Structure Type
          • Trade License Document Attachment
        • TL Service Configuration
          • Trade License Calculator
        • TL User Manual
          • Citizen User Manual
          • Employee User Manual
        • TL Demo Script
      • Property Tax
        • PT Brochure
        • PT Roadmap
        • PT Module Functional Specifications
        • PT Workflows
        • PT Implementation Guide
        • PT Master Data Templates
          • Mutation Fee
          • Rebate Rates
          • Penalty Rates
          • Interest Rates
          • Tax Rates
          • Unit Rates
          • Special Category Documents
          • Owner Special Category
          • Ownership Sub Category
          • Ownership Category
          • Usage Category Detail
          • Road Type
          • Construction Type
          • Property Type
          • Property Sub Type
          • Usage Category Major
          • Usage Category Minor
          • Usage Category Sub Minor
        • Property Tax Service
          • Property Mutation & Mutation Calculator
          • Assessment of Property
          • PT Calculator
        • PT - Create Property UI Details
          • Edit-Update Property
          • Property Tax - My Applications
          • Property Tax - My Properties
          • Property Tax - Quick Pay for Citizen
        • PT Data Migration
        • PT User Manual
          • Citizen User Manual
          • Employee User Manual
      • Water & Sewerage (W&S)
        • W&S Implementation Guide
        • W&S Module Functional Specifications
        • Water Services
          • Water Calculator Service
          • Sewerage Service
          • Sewerage Calculator Service
        • Sewerage Charges Master Data Templates
          • Sewerage Rates
          • Sewerage Penalty Rates
          • Sewerage Interest Rates
        • Water Charges Master Data Templates
          • Water Rates (Metered)
          • Pipe Size Types
          • Water Source Types
          • Water Rates (Non-Metered)
          • Water Penalty Rates
          • Water Interest Rates
        • W&S User Manual
          • W&S Citizen User Manual
          • W&S Employee User Manual
      • Public Grievances & Redressal (PGR)
        • PGR Brochure
        • PGR Roadmap
        • PGR Module Functional Specifications
        • PGR Workflows
        • PGR Implementation Guide
        • PGR Master Data Templates
          • Grievance Type
          • Grievance Sub Type
        • PGR Service Configuration
          • PGR Migration
        • PGR User Manual
          • Complaint Types List
          • Citizen User Manual
          • Employee User Manual
        • PGR Demo Script
      • Online Building Plan Approval System (OBPAS)
        • OBPAS Brochure
        • OBPAS Module Functional Specifications
        • OBPAS Implementation Guide
        • OBPAS Master Data Templates
          • Fee Structure
          • NOC Departments
          • Stakeholders Type
          • List Of Services
          • Service-Wise Documents
          • Building Occupancy
          • Building Sub Occupancy
          • Building Usage
          • Inspection Checklist
          • Town Planning Schemes
        • OBPAS Service Configuration
          • BPA Service Configuration
          • BPA Calculator Service
          • Setting Up eDCR Service
          • eDCR Integration
          • Noc Services
          • Land Services
        • OBPAS User Manual
          • OBPAS Citizen User Manual
          • OBPAS Employee User Manual
      • mCollect (MCS)
        • mCollect Roadmap
        • mCollect Master Data Templates
          • Service Category
          • Service Sub Category
          • Service Sub Category GL Code Mapping
        • eChallan UI Details
          • Edit - Cancel Challan
          • Search and Pay Challan
        • mCollect User Manual
          • MCS Citizen User Manual
          • MCS Employee User Manual
      • Faecal Sludge Management (FSM)
        • FSM Service Configuration
          • FSM Calculator v1.0
          • FSM Services
          • FSM Vendor Registry v1.0
          • FSM Vehicle Registry v1.0
        • FSM User Manual
          • Citizen User Manual
          • Employee User Manual
          • DSO User Manual
          • Septage Treatment Plant Operator User Manual
      • Finance
        • Finance Module Functional Specifications
        • Finance Implementation Guide
        • Finance Master Data Templates
          • Bank Account
          • Chart Of Accounts
          • Funds
          • Function
          • Contractors
          • Suppliers
          • Schemes
          • Sub Schemes
          • Banks
          • Bank Branch
          • Deduction
          • Opening Balances
          • Sub Ledger Category
          • Sub Ledger Master
        • Finance Service Configuration
        • Finance User Manual
          • Employee User Manual
          • Admin User Manual
          • Finance Reports Manual
      • Fire NOC
        • Fire NOC Master Data Templates
          • Building Usage Type
          • Building Sub Usage Type
          • Fire Station Master
          • Areas Served Master
          • Fire Station Mapping
          • Fire NOC Fee
        • Fire NOC Service
          • Fire NOC Calculator Service
        • Fire NOC User Manual
          • Fire NOC Citizen User Manual
          • Fire NOC Employee User Manual
      • e-Challan Service
        • eChallan Calculator Services
      • Product FAQs
  • Training Resources
    • Training Videos
    • Technical Enablement Areas
    • eDCR Training Plan
  • Partner Support
    • eGov Enablement Support for DIGIT
    • Security Guidelines Handbook
    • Troubleshooting Guide
  • Setup
    • Quickstart
    • Full Installation
      • On AWS
      • On Azure
      • On GCP
      • On SDC
      • On NIC
    • Open Source Tools used in DIGIT
    • Productionize DIGIT
      • scaling
        • Horizontally Scale DIGIT Services
    • Learn More
      • Setup Requirements
        • Tech Enablement Training - Essential Skills and Pre-requisites
        • Tech Enablement Training (eDCR) - Essential Skills and Prerequisites
        • DIGIT Rollout Program Governance
        • DevOps Skills Requirements
        • Infra Requirements
        • Team Composition for DIGIT Implementation
        • Infra Best Practices
        • Operational Best practices
        • Why Kubernetes for DIGIT
      • Supported Clouds
        • Google Cloud
        • Azure
        • AWS
        • VSphere
        • SDC
        • NIC
      • Deployment - Key Concepts
        • Security Practices
        • CI/CD
        • Readiness & Liveness
        • Resource Requests & Limits
      • Understanding ERP Stack
        • ERP Monolithic Architecture
        • ERP Hybrid Architecture
        • ERP Coexistence Architecture
        • APMDP-HYBRID-INFRA-ARCHITECTURE
        • eGov SmartCity eGovernance Suite
        • ERP Deployment Process
        • ERP Release Process
        • ERP User Guide
      • Deploying DIGIT Services
        • Deployment Architecture
        • Routing Traffic
        • Backbone Deployment
      • Troubleshooting
        • Distributed Tracing
        • Logging
        • Monitoring & Alerts
  • Configure
    • Configure DIGIT
      • Git Repos
      • Setting up DIGIT
        • Configuring InfraOps
        • Setting up DIGIT Environment
        • Email And SMS Setup
        • FileStore Setup
        • Setting Up SSL Certificate
        • Periodic Log Cleanup
      • Setting up Master Data
        • MDMS Overview
        • Configuring Tenants
        • Configuring Master Data
        • Adding New Master
        • State Level Vs City Level Master
      • Master Data Collection Templates
        • Environment Setup
          • State Level Setup
            • Tenants Information
            • SMS Account Configuration
            • Email Account Configuration
            • Google Play Store Account
            • Payment Gateway Configuration
            • POS Integration Configuration
            • Domain Name Configuration
            • SSL Configuration
            • ULB Departments
            • ULB Designations
            • Localization
            • Google Map Configuration
          • ULB Level Setup
            • Boundary Hierarchies
            • Boundary Data
            • Cross Hierarchy Mapping
            • ULB Bank Accounts
        • Module Setup
          • Web Portals Templates
            • State Portal
            • ULB Portal
          • HRMS Data Templates
            • User Roles
            • System Users
          • Billing And Payments Data Templates
            • Tax Heads
            • Receipt Format
            • Demand Bill Format
          • DSS Data Templates
            • KPI Acceptance
          • Workflow Data Templates
            • Workflow Actions
            • Workflow Levels
            • Workflow Process
            • Workflow Notifications
          • Common Configuration Details
            • Standard Document List
            • Service Document Mapping
            • Checklist
            • Configuring Data FAQs
      • Configuring DIGIT Service Stack
        • Core Services
          • Workflow Services
          • Location Services
          • User Services
          • Access Control Services
          • PDF Generation Service
          • MDMS (Master Data Management Service)
          • Payment Gateway Service
          • User Session Management In DIGIT
          • Indexer Service
          • URL Shortening Service
          • XState Core Chatbot
            • XState-Chatbot Integration Document
            • Xstate-Chatbot Message Localisation
        • Business Service
          • Appropriation Service
          • Billing Service
            • Bill Amendment Service Configuration
          • Collection Service
            • Collection Service Migration
          • Billing Collection Integration
          • Dashboard Analytics - Backend
          • DSS Technical Documentation
          • DSS Dashboard - Technical Document for UI
          • DSS Features Enhancements
          • Technical Script/Steps For Migration Process
          • Bill Amendment
        • Municipal Service
        • Utilities
      • Configuring Services
        • Persister Configuration
        • Indexer Configuration
        • Setting up a Language
          • Adding New Language
          • Setting Up Default Language For SMS & Emails
        • API Dos and Don'ts
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
        • Setting Up Service Locally
        • Configuring Workflows
          • Setting Up Workflows
          • Configuring Workflows For An Entity
        • DSS Configuration And Setup
          • Building New Dashboards
        • Configuring New Reports
          • Types Of Reports Used In Report Service
          • Impact Of Heavy Reports On Platform
        • Adding Roles To System
        • Mapping Roles With APIs
        • Deployment Of App on Play Store
        • Adding New APIs For Access
        • Configuring Localization
          • Setup Base Product Localization
          • Configure SMS and Email
        • Customizing PDF Notices And Certificates
          • Integration Of PDF In UI For Download And Print PDF
          • Customizing PDF Receipts & Certificates
      • QA Automation
        • Kafka Consumer Producer Approach
        • Jenkins Setup for Automation
        • Automation Framework Knowledge Base
        • Automation Test Tags
        • Automation Test Reporting
      • Configuration FAQs
  • Customize
    • Customize DIGIT
      • Frontend/UI
      • DIGIT Customization
        • API Do's & Don'ts
        • Writing A New Customer
        • Enhancing Existing Service
Powered by GitBook

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page
  • Introduction
  • Drift to Microservice Architecture
  • OpenTracing
  • Jaeger: A Distributed Tracing System by Uber
  • Major Components of Jaeger
  • Running Jaeger in a Docker Container
  • Creating Traces on Jaeger UI
  • Conclusion
  • References

Was this helpful?

Edit on Git
Export as PDF
  1. Setup
  2. Learn More
  3. Troubleshooting

Distributed Tracing

We will discuss distributed tracing system Jaeger and how it helps in troubleshooting DIGIT.

PreviousTroubleshootingNextLogging

Last updated 3 years ago

Was this helpful?

Introduction

Distributed tracing is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.

has been a key capability when it comes to microservices-based distributed systems like DIGIT. We’ll start with the introduction of OpenTracing, explaining what it is and why it is important We shall also set up Jaeger and learn to use it for monitoring and troubleshooting.

Drift to Microservice Architecture

Architecture has now become the obvious choice for application developers. In the Microservice Architecture, a monolithic application is broken down into a group of independently deployed services. In simple words, an application is more like a collection of microservices. When we have millions of such intertwined microservices working together, it’s almost impossible to map the inter-dependencies of these services and understand the execution of a request.

In case of a failure in a monolithic application, it is much easier to understand the path of a transaction and do the root cause analysis with the help of logging frameworks. But in a microservice architecture, logging alone fails to deliver the complete picture.

Is this service the first one in the call chain? How do I span all these services to get insight into the application? With questions like these, it becomes a significantly larger problem to debug a set of interdependent distributed services in comparison to a single monolithic application, making OpenTracing more and more popular.

OpenTracing

The OpenTracing API provides a standard, vendor-neutral framework for instrumentation. This means that if a developer wants to try out a different distributed tracing system, then instead of repeating the whole instrumentation process for the new distributed tracing system, the developer can simply change the configuration of the Tracer.

Here are some basic terminologies of Opentracing:

Span — It represents a logical unit of work that has an operation name, the start time of the operation, and the duration.

Trace — A Trace tells the story of a transaction or workflow as it propagates through a distributed system. It is simply a set of spans sharing a TraceID. Each component in a distributed system contributes its own span.

OpenTracing is a way for services to “describe and propagate distributed traces without knowledge of the underlying OpenTracing implementation.”

Let us take the example of a service like egov-property service (or any other DIGIT service). A service like this requires many other microservices to check that the location is available, proper payment credentials are received, and enough details exist for the ULB to process the property tac. If either one of those microservice fails, then the entire transaction fails. In such a case, having logs just for the main property service wouldn’t be very useful for debugging. However, if you were able to analyze each service you wouldn’t have to scratch your head to troubleshoot which microservice failed and what made it fail.

In real life, applications are even more complex and with the increasing complexity of applications, monitoring the applications has been a tedious task. Opentracing helps us to easily monitor:

  • Spans of services

  • Time taken by each service

  • Latency between the services

  • Hierarchy of services

  • Errors or exceptions during execution of each service.

Jaeger: A Distributed Tracing System by Uber

Jaeger is used for monitoring and troubleshooting microservices-based distributed systems, including:

  • Distributed transaction monitoring

  • Performance and latency optimization

  • Root cause analysis

  • Service dependency analysis

  • Distributed context propagation

Major Components of Jaeger

Agent — The Jaeger agent is a network daemon that listens for spans sent over UDP, which it batches and sends to the collector. It is designed to be deployed to all hosts as an infrastructure component. The agent abstracts the routing and discovery of the collectors away from the client.

Query — Query is a service that retrieves traces from storage and hosts a UI to display them.

Ingester — Ingester is a service that reads from Kafka topic and writes to another storage backend (Cassandra, Elasticsearch).

Running Jaeger in a Docker Container

  1. First, install Jaeger Client on your machine:

  2. Now, let’s run Jaeger backend as an all-in-one Docker image. The image launches the Jaeger UI, collector, query, and agent:

TIP: To check if the docker container is running, use: Docker ps.

Creating Traces on Jaeger UI

1. Create a Python program to create Traces

Let’s generate some traces using a simple python program. You can clone the Jaeger-Opentracing repository given below for a sample program that is used in this blog.

The Python program takes a movie name as an argument and calls three functions that get the cinema details, movie showtime details, and finally, book a movie ticket.

It creates some random delays in all the functions to make it more interesting, as, in reality, the functions would take a certain time to get the details. Also, the function throws random errors to give us a feel of how the traces of a real-life application may look like in case of failures.

Here is a brief description of how OpenTracing has been used in the program:

  • Initializing a tracer:

  • Using the tracer instance:

  • Starting new child spans using start_span:

  • Using Tags:

  • Using Logs:

2. Run the python program

Now, check your Jaeger UI, you can see a new service “booking” added. Select the service and click on “Find Traces” to see the traces of your service. Every time you run the program a new trace will be created.

You can now compare the duration of traces through the graph shown above. You can also filter traces using “Tags” section under “Find Traces”. For example, Setting “error=true” tag will filter out all the jobs that have errors.

To view the detailed trace, you can select a specific trace instance and check details like the time taken by each service, errors during execution and logs.

Conclusion

In this blog, we’ve described the importance and benefits of OpenTracing, one of the core pillars of modern applications. We also explored how distributed tracer Jaeger collect and store traces while revealing inefficient portions of our applications. It is fully compatible with OpenTracing API and has a number of clients for different programming languages including Java, Go, Node.js, Python, PHP, and more.

References

Jaeger Client Libraries — Jaeger clients are language-specific implementations of the .

Collector — The Jaeger collector receives traces from Jaeger agents and runs them through a processing pipeline. Currently, the pipeline validates traces, indexes them, performs transformations, and finally, stores them. Jaeger’s storage is a pluggable component which currently supports , , and .

Once the container starts, open to access the Jaeger UI. The container runs the Jaeger backend with an in-memory store, which is initially empty, so there is not much we can do with the UI right now since the store has no traces.

All content on this page by is licensed under a .

OpenTracing API
Cassandra
Elasticsearch
Kafka
http://localhost:16686/
https://www.jaegertracing.io/docs/1.9/
https://opentracing.io/docs/
OpenTracing
Microservice
​
eGov Foundation
Creative Commons Attribution 4.0 International License
Creative Commons License