DIGIT Urban
PlatformDomainsAcademyDesign SystemFeedback
v2.7
v2.7
  • Introducing DIGIT Urban
  • Platform
    • Release Notes
      • MDMS Configuration & Service Build Updates
      • Test Cases
      • FSM Release Notes v1.1
        • FSM Service Build Updates
        • FSM MDMS Configuration Updates
        • FSM Test Cases
        • FSM Module Release Notes
      • National DSS & State DSS Release Notes
      • Birth & Death Release Notes
      • Common Features UI/UX Revamp
      • Property Tax Release Notes
    • Platform Features
    • Architecture
    • Specifications
    • Installation
      • Setup Basics
        • Setup Requirements
          • Tech Enablement Training - Essential Skills and Pre-requisites
          • Tech Enablement Training (eDCR) - Essential Skills and Prerequisites
          • DIGIT Rollout Program Governance
          • DevOps Skills Requirements
          • Infra Requirements
          • Team Composition for DIGIT Implementation
          • Infra Best Practices
          • Operational Best practices
          • Why Kubernetes for DIGIT
        • Supported Clouds
          • Google Cloud
          • Azure
          • AWS
          • VSphere
          • SDC
          • NIC
        • Deployment - Key Concepts
          • Security Practices
          • CI/CD
          • Readiness & Liveness
          • Resource Requests & Limits
        • Understanding ERP Stack
          • ERP Monolithic Architecture
          • ERP Hybrid Architecture
          • ERP Coexistence Architecture
          • APMDP-HYBRID-INFRA-ARCHITECTURE
          • eGov SmartCity eGovernance Suite
          • ERP Deployment Process
          • ERP Release Process
          • ERP User Guide
        • Deploying DIGIT Services
          • Deployment Architecture
          • Routing Traffic
          • Backbone Deployment
        • Troubleshooting
          • Distributed Tracing
          • Logging
          • Monitoring & Alerts
    • Configuration
      • Git Repos
      • Setting up Master Data
        • MDMS Overview
        • Configuring Tenants
        • Configuring Master Data
        • Adding New Master
        • State Level Vs City Level Master
      • Master Data Collection Templates
        • Environment Setup
          • State Level Setup
            • Tenants Information
            • SMS Account Configuration
            • Email Account Configuration
            • Google Play Store Account
            • Payment Gateway Configuration
            • POS Integration Configuration
            • Domain Name Configuration
            • SSL Configuration
            • ULB Departments
            • ULB Designations
            • Localization
            • Google Map Configuration
          • ULB Level Setup
            • Boundary Hierarchies
            • Boundary Data
            • Cross Hierarchy Mapping
            • ULB Bank Accounts
        • Module Setup
          • Web Portals Templates
            • State Portal
            • ULB Portal
          • HRMS Data Templates
            • User Roles
            • System Users
          • Billing And Payments Data Templates
            • Tax Heads
            • Receipt Format
            • Demand Bill Format
          • DSS Data Templates
            • KPI Acceptance
          • Workflow Data Templates
            • Workflow Actions
            • Workflow Levels
            • Workflow Process
            • Workflow Notifications
          • Common Configuration Details
            • Standard Document List
            • Service Document Mapping
            • Checklist
            • Configuring Data FAQs
      • Configuring DIGIT Service Stack
        • Core Services
          • Workflow Services
          • Location Services
          • User Services
          • Access Control Services
          • PDF Generation Service
          • MDMS (Master Data Management Service)
          • Payment Gateway Service
          • NLP Engine Service
          • User Session Management In DIGIT
          • Indexer Service
          • URL Shortening Service
          • XState Core Chatbot
            • XState-Chatbot Integration Document
            • Xstate-Chatbot Message Localisation
        • Business Service
          • Appropriation Service
          • Billing Service
            • Bill Amendment Service Configuration
          • Collection Service
            • Collection Service Migration
            • Collection Service V2
          • Billing Collection Integration
          • Dashboard Analytics - Backend
            • DSS Features Enhancements
            • DSS Dashboard - Technical Document for UI
            • DSS Technical Documentation
          • Technical Script/Steps For Migration Process
        • Municipal Service
          • Turn-Io-Adapter
          • Inbox Service
        • Utilities
      • Configuring Services
        • Persister Configuration
        • Indexer Configuration
        • Setting up a Language
          • Adding New Language
          • Setting Up Default Language For SMS & Emails
        • API Checklist
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
        • Setting Up Service Locally
        • Configuring Workflows
          • Setting Up Workflows
          • Configuring Workflows For An Entity
          • Workflow Auto Escalation
        • DSS Configuration And Setup
          • Building New Dashboards
        • Configuring New Reports
          • Types Of Reports Used In Report Service
          • Impact Of Heavy Reports On Platform
        • Adding Roles To System
        • Mapping Roles With APIs
        • Deployment Of App on Play Store
        • Adding New APIs For Access
        • FileStore Service
        • Telemetry Service
        • Report Service
        • ID Generation Service
        • Localization Service
        • Email Notification Service
        • SMS Notification Service
        • SMS Template Approval Process
        • Searcher Service
        • Zuul Service
        • User OTP Service
        • eGov OTP Service
        • Persister Service
        • Encryption Service
        • Document Uploader Service
        • Notification Enhancement Based on Different Channels
        • Configuring Localization
          • Setup Base Product Localization
          • Configure SMS and Email
        • DIGIT: Internal Datamart Deployment Steps
        • Customizing PDF Notices And Certificates
          • Integration Of PDF In UI For Download And Print PDF
          • Customizing PDF Receipts & Certificates
      • QA Automation
        • Kafka Consumer Producer Approach
        • Jenkins Setup for Automation
        • Automation Framework Knowledge Base
          • Automation Test Tags
          • Automation Test Reporting
        • State-Level UI Customization
      • Configuration FAQs
    • Customization
      • Frontend/UI
      • DIGIT Customization
        • API Do's & Don'ts
        • Writing A New Customer
        • Enhancing Existing Service
  • Products
    • Products & Modules
      • Trade Licence (TL)
        • TL Brochure
        • TL Module Functional Specifications
        • TL Implementation Guide
        • TL Master Data Templates
          • Trade Type
          • Trade Sub Type
          • Trade Category
          • Trade License Fee
          • Structure Type
          • Structure Sub Type
          • Trade License Document Attachment
        • TL Service Configuration
          • Trade License Calculator
          • TL Apply Flow UI Details
            • My Applications UI Flow
            • Send Back - Edit UI Flow
            • Trade License Renewal UI Flow
            • Employee Inbox UI Details
            • Employee Search Application Search License UI Flow
            • New Trade License UI Flow
            • Application Details - Trade Details UI Flows
            • Renew Edit Application
        • TL User Manual
          • TL Citizen User Manual
          • TL Employee User Manual
        • TL Demo Script
      • Property Tax (PT)
        • PT Brochure
        • PT Module Functional Specifications
        • PT Implementation Guide
        • PT Master Data Templates
          • Mutation Fee
          • Rebate Rates
          • Penalty Rates
          • Interest Rates
          • Tax Rates
          • Unit Rates
          • Special Category Documents
          • Owner Special Category
          • Ownership Sub Category
          • Ownership Category
          • Usage Category Detail
          • Road Type
          • Construction Type
          • Property Type
          • Property Sub Type
          • Usage Category Major
          • Usage Category Minor
          • Usage Category Sub Minor
        • PT Service Configuration
          • Property Mutation & Mutation Calculator
          • PT Update Number & Alternate Number
            • Update Primary Mobile Number & Alternate Mobile Number
          • Assessment of Property
          • PT Calculator
          • Fuzzy Search
            • Fuzzy Search Reindexing
          • Common PT
          • PT - Create Property UI Details
            • Edit-Update Property
            • Property Tax - My Applications
            • Property Tax - My Properties
            • Property Tax - Search Applications
            • Property Tax - Quick Pay for Citizen
            • Property Tax - Search And Pay My Bills
            • Property Tax - My Payments
            • Create Application - Employee UI/UX Revamp
            • Employee Edit Application Flow
            • Employee - Search Property, Property Details Page & Assessment
            • Employee - Inbox & Application Details
            • Employee Inbox (Old UI)
            • Citizen Mutation Flow
            • Employee - Mutation (Ownership Transfer)
          • PT Data Migration
        • PT User Manual
          • PT Citizen User Manual
          • PT Employee User Manual
      • Water & Sewerage (W&S)
        • W&S Brochure
        • W&S Module Functional Specifications
        • W&S Implementation Guide
        • Water Service Configuration
          • Legacy/Re-Indexing the Water & Sewerage Service
          • State DSS - Water & Sewerage
          • Water Calculator Service
          • Sewerage Service Configuration
            • Sewerage Calculator Service
        • Sewerage Charges Master Data Templates
          • Sewerage Rates
          • Sewerage Penalty Rates
          • Sewerage Interest Rates
        • Water Charges Master Data Templates
          • Water Rates (Metered)
          • Pipe Size Types
          • Water Source Types
          • Water Rates (Non-Metered)
          • Water Penalty Rates
          • Water Interest Rates
        • W&S User Manual
          • W&S Citizen User Manual
          • W&S Employee User Manual
      • Public Grievances & Redressal (PGR)
        • PGR Brochure
        • PGR Module Functional Specifications
        • PGR Workflows
        • PGR Master Data Templates
          • Grievance Type
          • Grievance Sub Type
          • Routing Matrix
          • Escalation Matrix
        • PGR Service Configuration
          • PGR Migration
        • PGR User Manual
          • Complaint Types List
          • PGR Citizen User Manual
          • PGR Employee User Manual
        • PGR Demo Script
      • Online Building Plan Approval System (OBPAS)
        • OBPAS Brochure
        • OBPAS Module Functional Specifications
        • OBPAS Implementation Guide
        • OBPAS Master Data Templates
          • Fee Structure
          • NOC Departments
          • Stakeholders Type
          • List Of Services
          • Service-Wise Documents
          • Building Occupancy
          • Building Sub Occupancy
          • Building Usage
          • Inspection Checklist
          • Town Planning Schemes
        • OBPAS Service Configuration
          • BPA Service Configuration
          • BPA Stakeholder Registration
          • Legacy/Re-Indexing For OBPS
          • BPA Calculator Service
          • Setting Up eDCR Service
          • eDCR Integration
          • Noc Services
          • Land Services
          • State DSS - OBPS
          • OBPAS DIGIT UI
            • Stakeholder - Registration Flow
            • OBPS-BPA/OC-BPA Employee Flow
              • OBPS Inbox
              • OBPS Search Application
            • OBPS-BPA/OC-BPA Architect
            • OBPS EDCR /OC-EDCR
            • OBPS BPA / OCBPA Citizen
            • Employee - Stakeholder
              • Stakeholder Inbox
              • Stakeholder Search Application
        • OBPAS User Manual
          • OBPAS Citizen User Manual
          • OBPAS Employee User Manual
        • OBPAS Demo Script
      • mCollect (MCS)
        • MCS Functional Specifications
        • mCollect Master Data Templates
          • Service Category
          • Service Sub Category
          • Service Sub Category GL Code Mapping
        • MCS Service Configuration
          • e-Challan Service
            • e-Challan Calculator Service
          • Re-Indexing The mCollect Service
          • State DSS - mCollect
          • Current Bill Cancellation UI Flow
            • Bill Details UI Flow
            • Cancel Bill UI Flow
          • Receipt Cancellation UI Flow
            • View Receipt - Cancel UI Flow
          • e-Challan UI Details
            • Edit - Cancel Challan
            • Search and Pay Challan
            • mCollect UI Flow
            • Update - Cancel Challan UI Flow
            • Challan Creation
        • mCollect User Manual
          • MCS Citizen User Manual
          • MCS Employee User Manual
      • Faecal Sludge Management (FSM)
      • Finance
        • Finance Module Functional Specifications
        • Finance Implementation Guide
        • Finance Master Data Templates
          • Bank Account
          • Chart Of Accounts
          • Funds
          • Function
          • Contractors
          • Suppliers
          • Schemes
          • Sub Schemes
          • Banks
          • Bank Branch
          • Deduction
          • Opening Balances
          • Sub Ledger Category
          • Sub Ledger Master
        • Finance Service Configuration
          • Finance Collections Voucher Consumer Service
        • Finance User Manual
          • Finance Employee User Manual
          • Admin User Manual
          • Finance Reports Manual
      • Fire NOC
        • Fire NOC Functional Specifications
        • Fire NOC Master Data Templates
          • Building Usage Type
          • Building Sub Usage Type
          • Fire Station Master
          • Areas Served Master
          • Fire Station Mapping
          • Fire NOC Fee
        • Fire NOC Service
          • Re-Indexing the Fire NOC Service
          • State DSS - Fire NOC
          • Fire NOC Calculator Service
          • NOC DIGIT UI
            • NOC Inbox
            • Application Overview
        • Fire NOC User Manual
          • Fire NOC Citizen User Manual
          • Fire NOC Employee User Manual
      • HRMS
        • HRMS Functional Specifications
        • HRMS Employee Create-Edit UI Flow
        • Employee Details UI Flow
        • Employee Activation-Deactivation UI Flow
        • Search Employee By Multiple Criteria UI Flow
        • HRMS Service
        • Employees Count UI Flow
        • Auto Escalation UI Flow
      • Events And Notifications
        • User Events Services
        • Citizen Flow
        • Employee Flow - Events
        • Employee Flow - Messages
      • Citizen Engagement Module
        • Document Uploader Service
        • Documents: Implementation Details
      • Common UI Docs
        • User Profile
          • Citizen Profile
          • Employee Profile
          • Employee City Change
        • Update Number & Invalid Number
        • Troubleshoot Using Browser Network Tab
      • Birth & Death
        • Birth & Death User Manual
          • Birth & Death Citizen User Manual
          • Birth & Death Employee User Manual
        • Birth & Death Service Configuration
          • State DSS - Birth and Death
          • Birth & Death Service Setup
      • National Urban Real-Time Dashboard
        • National Urban Dashboard User Manual
        • National Urban Dashboard Functional Specifications
        • National Dashboard Service Configuration
          • National Dashboard - UI Technical Doc
          • DSS UI Flow
          • W&S - National Dashboard
          • Property Tax - National Dashboard
          • Fire NOC - State DSS
          • National Dashboard - Overview
          • National Urban Realtime Dashboard
          • Trade License - National Dashboard
          • OBPS - National Dashboard
          • PGR - National Dashboard
          • National Dashboard Ingest Service
          • National Dashboard API Performance Testing and Benchmark
          • National Dashboard: Index Creation Steps
          • mCollect National Dashboard
      • Product FAQs
    • Training Resources
      • Training Videos
      • Technical Enablement Areas
      • eDCR Training Plan
    • Partner Support
      • eGov Enablement Support for DIGIT
      • Security Guidelines Handbook
      • Troubleshooting Guide
  • 🛃Ecosystem
    • DIGIT Pre-Sales Toolkit
      • Infra Calculator
      • Resource Calculator
      • DIGIT Services
    • Access DIGIT
    • Urban Docs Guide
    • Roadmap
    • Open Events
    • Source Code
Powered by GitBook

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page
  • Overview
  • Pre-requisites
  • Key Functionalities
  • Reference Docs

Was this helpful?

Edit on GitHub
Export as PDF
  1. Platform
  2. Configuration
  3. Configuring Services

DSS Configuration And Setup

Overview

A decision support system (DSS) is a composite tool that collects, organizes, and analyzes business data to facilitate quality decision-making for management, operations, and planning. A well-designed DSS aids decision-makers in compiling a variety of data from many sources: raw data, documents, and personal knowledge from employees, management, executives, and business models. DSS analysis helps organizations identify and solve problems, and make decisions

This document explains the steps on how to define the configurations & set up the new dashboard in the DSS.

Pre-requisites

Before you proceed with the configuration, make sure the following pre-requisites are met -

  • Prior knowledge of Spring boot

  • Prior knowledge of Kafka

  • Prior knowledge of Elastic Search

  • Prior knowledge of Kibana

  • Prior knowledge of EQL (Elastic Query Language)

  • Prior knowledge of JSON

Key Functionalities

  1. Creating a DSS dashboard schema

  2. DSS ingest service APIs

  3. Ingest service configurations

  4. Creating a Kafka sync connector to push the data to elastic search

1. Creating a DSS dashboard schema

When we are going to start indexing the DSS collection v2 index. We should create the schema in the ES using the Kibana query as there in the below file.

2. DSS ingest service API

curl -X POST \
http://dashboard-ingest.egov:8080/dashboard-ingest/ingest/migrate/paymentsindex-v1/v1 \
   -H 'Cache-Control: no-cache' \
   -H 'Content-Type: application/json' \
   -H 'Postman-Token: d380bebb-383f-1b7c-76d1-10c1dc07dd06' \
   -d '{
   "RequestInfo": {
     "apiId": "string",
     "ver": "string",
     "ts": null,
     "action": "string",
     "did": "string",
     "key": "string",
     "msgId": "string",
     "authToken": "b843ef27-1ac6-49b8-ab71-cd0c22f4e50e",
     "userInfo": {
       "id": 23299,
       "uuid":"e721639b-c095-40b3-86e2-acecb2cb6efb",
       "userName": "9337682030",
       "name": "Abhilash Seth",
       "type": "CITIZEN",
       "mobileNumber": "9337682030",
       "emailId": "abhilash.seth@gmail.com",
       "roles": [
         {
           "id": 281,
           "name": "Citizen"
         }
       ]
     }
   }
}'

3. Ingest service configurations

  • Transform collection schema for V2

    • This transform collection v1 configuration file is used to map the incoming data. This mapped data will go inside the data object in the DSS collection v2 index.

  • Here: $i, the variable value that gets incremented for the number of records of paymentDetails.

    $j, the variable value that gets incremented for the number of records of billDetails.

  • Enrichment Domain Configuration

    • This configuration defines and directs the Enrichment Process that the data goes through.

    • For example, if the data which is incoming is belonging to a Collection Module data, then the Collection Domain config is picked. And based on the Business Type specified in the data, the right config is picked.

    • In order to enhance the data of Collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set.

  • Topic Context Configuration

    • Topic Context Configuration is an outline to define which data is received on which Kafka Topic.

    • Indexer Service and many other services are sending out data on different Kafka Topics. If the Ingest Service is asked to receive those data and pass it through the pipeline, the context and the version of the data being received have to be set. This configuration is used to identify as in which Kafka topic consumed the data and what is the mapping for that.

  • JOLT Domain Transformation Schema

    • JOLT is a JSON to JSON Transformation Library. In order to change the structure of the data and transform it in a generic way, JOLT has been used.

    • While the transformation schemas are written for each Data Context, the data is transformed against the schema to obtain transformed data.

  • Validator Schema

    • Validator Schema is a configuration Schema Library from Everit Bypassing the data against this schema, it ensures that the data abides by the rules and requirements of the schema which has been defined.

  • Enhance Domain configuration

    • This configuration defines and directs the Enrichment Process that the data goes through.

    • For example, if the Data which is incoming is belonging to a Collection Module data, then the Collection Domain Config is picked. And based on the Business Type specified in the data, the right config is picked and the final data is placed inside the domain object.

    • In order to enhance the data of Collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set.

For the Kafka connect to work, Ingest pipeline application properties or in environments direct push must be disabled.

es.push.direct=false

If DSS collection index data is indexing directly ( without Kafka connector) to ES through the ingest pipeline then, make the application properties or in environments, direct push must be enabled.

es.push.direct=true

4. Creating a Kafka sync connector to push the data to the Elasticsearch

  • Configure the Kafka topics in the environments or Ingest pipeline application properties as shown below.

  • To start the indexing we will create a connecter that will take data from the topic and push it to the index we have mentioned in the "transforms.TopicNameRouter.replacement" and mention the ES host in the Kafka connection we have to mention the host URL in “connection.url”.

  • To create the Kafka connector run the below curl command inside the playground pod:

curl -X POST \
 http://kafka-connect.kafka-cluster:8083/connectors/ \
 -H 'Cache-Control: no-cache' \
 -H 'Content-Type: application/json' \
 -H 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
 -d '{
     "name": "dss-collection_v2-es-sink",
     "config": {
	 "connector.class":
"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
	 "type.name": "general",
	 "tasks.max": "1",
	 "max.retries": "15",
	 "key.ignore": "false",
	 "retry.backoff.ms": "5000",
	 "max.buffered.records": "2000",
	 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
	 "errors.log.enable": "true",
	 "key.converter":
"org.apache.kafka.connect.storage.StringConverter",
	 "read.timeout.ms": "10000",
	 "topics": "dss-collection-update",
	 "batch.size": "1000",
	 "max.in.flight.requests": "2",
	 "schema.ignore": "true",
	 "behavior.on.malformed.documents": "warn",
	 "flush.timeout.ms": "3600000",
	 "errors.deadletterqueue.topic.name": "dss-collection_v2-failed",
	 "errors.tolerance": "all",
	 "value.converter.schemas.enable": "false",
	 "name": "dss-collection_v2-es-sink",
	 "connection.url": "http://elasticsearch-client-v1.es-cluster:9200",
	 "linger.ms": "1000",
	 "transforms": "TopicNameRouter",
	 "transforms.TopicNameRouter.type":
"org.apache.kafka.connect.transforms.RegexRouter",
	 "transforms.TopicNameRouter.regex": "dss-collection_v2*",
	 "transforms.TopicNameRouter.replacement": "dss-collection_v2"
     }
}'
curl --location --request DELETE 'http://kafka-connect.kafka-cluster:8080/connectors/dss-collection_v2-es-sink'

Reference Docs

Doc Links

Description
Link

DSS Backend Configuration Manual

DSS Dashboard - Technical Document for UI

DSS Technical Documentation

PreviousWorkflow Auto EscalationNextBuilding New Dashboards

Last updated 2 years ago

Was this helpful?

Micro Service which runs as a pipeline and manages to validate, transform and enrich the incoming data and pushes the same to ElasticSearch Index. Ingest service will fetch the data from the index (paymentsindex-v1) which is specified in the indexing service API as below. The ingest service will read the configuration files which are there with v1. All the configuration files will be there .

All content on this page by is licensed under a .

here
Click here for an example configuration
Click here for an example configuration
Click here for an example configuration
Follow the slide deck for JOLT Transformations
Click here for an example configuration
Click here for an example configuration
Click here for an example configuration
DSS Backend Configuration Manual
https://digit-discuss.atlassian.net/wiki/spaces/EPE/pages/283017217/DSS+Dashboard+-+Technical+Document+for+UI
DSS Technical Documentation
​
eGov Foundation
Creative Commons Attribution 4.0 International License
Creative Commons License