DIGIT Urban
PlatformDomainsAcademyDesign SystemFeedback
v2.7
v2.7
  • Introducing DIGIT Urban
  • Platform
    • Release Notes
      • MDMS Configuration & Service Build Updates
      • Test Cases
      • FSM Release Notes v1.1
        • FSM Service Build Updates
        • FSM MDMS Configuration Updates
        • FSM Test Cases
        • FSM Module Release Notes
      • National DSS & State DSS Release Notes
      • Birth & Death Release Notes
      • Common Features UI/UX Revamp
      • Property Tax Release Notes
    • Platform Features
    • Architecture
    • Specifications
    • Installation
      • Setup Basics
        • Setup Requirements
          • Tech Enablement Training - Essential Skills and Pre-requisites
          • Tech Enablement Training (eDCR) - Essential Skills and Prerequisites
          • DIGIT Rollout Program Governance
          • DevOps Skills Requirements
          • Infra Requirements
          • Team Composition for DIGIT Implementation
          • Infra Best Practices
          • Operational Best practices
          • Why Kubernetes for DIGIT
        • Supported Clouds
          • Google Cloud
          • Azure
          • AWS
          • VSphere
          • SDC
          • NIC
        • Deployment - Key Concepts
          • Security Practices
          • CI/CD
          • Readiness & Liveness
          • Resource Requests & Limits
        • Understanding ERP Stack
          • ERP Monolithic Architecture
          • ERP Hybrid Architecture
          • ERP Coexistence Architecture
          • APMDP-HYBRID-INFRA-ARCHITECTURE
          • eGov SmartCity eGovernance Suite
          • ERP Deployment Process
          • ERP Release Process
          • ERP User Guide
        • Deploying DIGIT Services
          • Deployment Architecture
          • Routing Traffic
          • Backbone Deployment
        • Troubleshooting
          • Distributed Tracing
          • Logging
          • Monitoring & Alerts
    • Configuration
      • Git Repos
      • Setting up Master Data
        • MDMS Overview
        • Configuring Tenants
        • Configuring Master Data
        • Adding New Master
        • State Level Vs City Level Master
      • Master Data Collection Templates
        • Environment Setup
          • State Level Setup
            • Tenants Information
            • SMS Account Configuration
            • Email Account Configuration
            • Google Play Store Account
            • Payment Gateway Configuration
            • POS Integration Configuration
            • Domain Name Configuration
            • SSL Configuration
            • ULB Departments
            • ULB Designations
            • Localization
            • Google Map Configuration
          • ULB Level Setup
            • Boundary Hierarchies
            • Boundary Data
            • Cross Hierarchy Mapping
            • ULB Bank Accounts
        • Module Setup
          • Web Portals Templates
            • State Portal
            • ULB Portal
          • HRMS Data Templates
            • User Roles
            • System Users
          • Billing And Payments Data Templates
            • Tax Heads
            • Receipt Format
            • Demand Bill Format
          • DSS Data Templates
            • KPI Acceptance
          • Workflow Data Templates
            • Workflow Actions
            • Workflow Levels
            • Workflow Process
            • Workflow Notifications
          • Common Configuration Details
            • Standard Document List
            • Service Document Mapping
            • Checklist
            • Configuring Data FAQs
      • Configuring DIGIT Service Stack
        • Core Services
          • Workflow Services
          • Location Services
          • User Services
          • Access Control Services
          • PDF Generation Service
          • MDMS (Master Data Management Service)
          • Payment Gateway Service
          • NLP Engine Service
          • User Session Management In DIGIT
          • Indexer Service
          • URL Shortening Service
          • XState Core Chatbot
            • XState-Chatbot Integration Document
            • Xstate-Chatbot Message Localisation
        • Business Service
          • Appropriation Service
          • Billing Service
            • Bill Amendment Service Configuration
          • Collection Service
            • Collection Service Migration
            • Collection Service V2
          • Billing Collection Integration
          • Dashboard Analytics - Backend
            • DSS Features Enhancements
            • DSS Dashboard - Technical Document for UI
            • DSS Technical Documentation
          • Technical Script/Steps For Migration Process
        • Municipal Service
          • Turn-Io-Adapter
          • Inbox Service
        • Utilities
      • Configuring Services
        • Persister Configuration
        • Indexer Configuration
        • Setting up a Language
          • Adding New Language
          • Setting Up Default Language For SMS & Emails
        • API Checklist
        • Setting Up SMS Gateway
          • Using The Generic GET & POST SMS Gateway Interface
        • Setting Up Service Locally
        • Configuring Workflows
          • Setting Up Workflows
          • Configuring Workflows For An Entity
          • Workflow Auto Escalation
        • DSS Configuration And Setup
          • Building New Dashboards
        • Configuring New Reports
          • Types Of Reports Used In Report Service
          • Impact Of Heavy Reports On Platform
        • Adding Roles To System
        • Mapping Roles With APIs
        • Deployment Of App on Play Store
        • Adding New APIs For Access
        • FileStore Service
        • Telemetry Service
        • Report Service
        • ID Generation Service
        • Localization Service
        • Email Notification Service
        • SMS Notification Service
        • SMS Template Approval Process
        • Searcher Service
        • Zuul Service
        • User OTP Service
        • eGov OTP Service
        • Persister Service
        • Encryption Service
        • Document Uploader Service
        • Notification Enhancement Based on Different Channels
        • Configuring Localization
          • Setup Base Product Localization
          • Configure SMS and Email
        • DIGIT: Internal Datamart Deployment Steps
        • Customizing PDF Notices And Certificates
          • Integration Of PDF In UI For Download And Print PDF
          • Customizing PDF Receipts & Certificates
      • QA Automation
        • Kafka Consumer Producer Approach
        • Jenkins Setup for Automation
        • Automation Framework Knowledge Base
          • Automation Test Tags
          • Automation Test Reporting
        • State-Level UI Customization
      • Configuration FAQs
    • Customization
      • Frontend/UI
      • DIGIT Customization
        • API Do's & Don'ts
        • Writing A New Customer
        • Enhancing Existing Service
  • Products
    • Products & Modules
      • Trade Licence (TL)
        • TL Brochure
        • TL Module Functional Specifications
        • TL Implementation Guide
        • TL Master Data Templates
          • Trade Type
          • Trade Sub Type
          • Trade Category
          • Trade License Fee
          • Structure Type
          • Structure Sub Type
          • Trade License Document Attachment
        • TL Service Configuration
          • Trade License Calculator
          • TL Apply Flow UI Details
            • My Applications UI Flow
            • Send Back - Edit UI Flow
            • Trade License Renewal UI Flow
            • Employee Inbox UI Details
            • Employee Search Application Search License UI Flow
            • New Trade License UI Flow
            • Application Details - Trade Details UI Flows
            • Renew Edit Application
        • TL User Manual
          • TL Citizen User Manual
          • TL Employee User Manual
        • TL Demo Script
      • Property Tax (PT)
        • PT Brochure
        • PT Module Functional Specifications
        • PT Implementation Guide
        • PT Master Data Templates
          • Mutation Fee
          • Rebate Rates
          • Penalty Rates
          • Interest Rates
          • Tax Rates
          • Unit Rates
          • Special Category Documents
          • Owner Special Category
          • Ownership Sub Category
          • Ownership Category
          • Usage Category Detail
          • Road Type
          • Construction Type
          • Property Type
          • Property Sub Type
          • Usage Category Major
          • Usage Category Minor
          • Usage Category Sub Minor
        • PT Service Configuration
          • Property Mutation & Mutation Calculator
          • PT Update Number & Alternate Number
            • Update Primary Mobile Number & Alternate Mobile Number
          • Assessment of Property
          • PT Calculator
          • Fuzzy Search
            • Fuzzy Search Reindexing
          • Common PT
          • PT - Create Property UI Details
            • Edit-Update Property
            • Property Tax - My Applications
            • Property Tax - My Properties
            • Property Tax - Search Applications
            • Property Tax - Quick Pay for Citizen
            • Property Tax - Search And Pay My Bills
            • Property Tax - My Payments
            • Create Application - Employee UI/UX Revamp
            • Employee Edit Application Flow
            • Employee - Search Property, Property Details Page & Assessment
            • Employee - Inbox & Application Details
            • Employee Inbox (Old UI)
            • Citizen Mutation Flow
            • Employee - Mutation (Ownership Transfer)
          • PT Data Migration
        • PT User Manual
          • PT Citizen User Manual
          • PT Employee User Manual
      • Water & Sewerage (W&S)
        • W&S Brochure
        • W&S Module Functional Specifications
        • W&S Implementation Guide
        • Water Service Configuration
          • Legacy/Re-Indexing the Water & Sewerage Service
          • State DSS - Water & Sewerage
          • Water Calculator Service
          • Sewerage Service Configuration
            • Sewerage Calculator Service
        • Sewerage Charges Master Data Templates
          • Sewerage Rates
          • Sewerage Penalty Rates
          • Sewerage Interest Rates
        • Water Charges Master Data Templates
          • Water Rates (Metered)
          • Pipe Size Types
          • Water Source Types
          • Water Rates (Non-Metered)
          • Water Penalty Rates
          • Water Interest Rates
        • W&S User Manual
          • W&S Citizen User Manual
          • W&S Employee User Manual
      • Public Grievances & Redressal (PGR)
        • PGR Brochure
        • PGR Module Functional Specifications
        • PGR Workflows
        • PGR Master Data Templates
          • Grievance Type
          • Grievance Sub Type
          • Routing Matrix
          • Escalation Matrix
        • PGR Service Configuration
          • PGR Migration
        • PGR User Manual
          • Complaint Types List
          • PGR Citizen User Manual
          • PGR Employee User Manual
        • PGR Demo Script
      • Online Building Plan Approval System (OBPAS)
        • OBPAS Brochure
        • OBPAS Module Functional Specifications
        • OBPAS Implementation Guide
        • OBPAS Master Data Templates
          • Fee Structure
          • NOC Departments
          • Stakeholders Type
          • List Of Services
          • Service-Wise Documents
          • Building Occupancy
          • Building Sub Occupancy
          • Building Usage
          • Inspection Checklist
          • Town Planning Schemes
        • OBPAS Service Configuration
          • BPA Service Configuration
          • BPA Stakeholder Registration
          • Legacy/Re-Indexing For OBPS
          • BPA Calculator Service
          • Setting Up eDCR Service
          • eDCR Integration
          • Noc Services
          • Land Services
          • State DSS - OBPS
          • OBPAS DIGIT UI
            • Stakeholder - Registration Flow
            • OBPS-BPA/OC-BPA Employee Flow
              • OBPS Inbox
              • OBPS Search Application
            • OBPS-BPA/OC-BPA Architect
            • OBPS EDCR /OC-EDCR
            • OBPS BPA / OCBPA Citizen
            • Employee - Stakeholder
              • Stakeholder Inbox
              • Stakeholder Search Application
        • OBPAS User Manual
          • OBPAS Citizen User Manual
          • OBPAS Employee User Manual
        • OBPAS Demo Script
      • mCollect (MCS)
        • MCS Functional Specifications
        • mCollect Master Data Templates
          • Service Category
          • Service Sub Category
          • Service Sub Category GL Code Mapping
        • MCS Service Configuration
          • e-Challan Service
            • e-Challan Calculator Service
          • Re-Indexing The mCollect Service
          • State DSS - mCollect
          • Current Bill Cancellation UI Flow
            • Bill Details UI Flow
            • Cancel Bill UI Flow
          • Receipt Cancellation UI Flow
            • View Receipt - Cancel UI Flow
          • e-Challan UI Details
            • Edit - Cancel Challan
            • Search and Pay Challan
            • mCollect UI Flow
            • Update - Cancel Challan UI Flow
            • Challan Creation
        • mCollect User Manual
          • MCS Citizen User Manual
          • MCS Employee User Manual
      • Faecal Sludge Management (FSM)
      • Finance
        • Finance Module Functional Specifications
        • Finance Implementation Guide
        • Finance Master Data Templates
          • Bank Account
          • Chart Of Accounts
          • Funds
          • Function
          • Contractors
          • Suppliers
          • Schemes
          • Sub Schemes
          • Banks
          • Bank Branch
          • Deduction
          • Opening Balances
          • Sub Ledger Category
          • Sub Ledger Master
        • Finance Service Configuration
          • Finance Collections Voucher Consumer Service
        • Finance User Manual
          • Finance Employee User Manual
          • Admin User Manual
          • Finance Reports Manual
      • Fire NOC
        • Fire NOC Functional Specifications
        • Fire NOC Master Data Templates
          • Building Usage Type
          • Building Sub Usage Type
          • Fire Station Master
          • Areas Served Master
          • Fire Station Mapping
          • Fire NOC Fee
        • Fire NOC Service
          • Re-Indexing the Fire NOC Service
          • State DSS - Fire NOC
          • Fire NOC Calculator Service
          • NOC DIGIT UI
            • NOC Inbox
            • Application Overview
        • Fire NOC User Manual
          • Fire NOC Citizen User Manual
          • Fire NOC Employee User Manual
      • HRMS
        • HRMS Functional Specifications
        • HRMS Employee Create-Edit UI Flow
        • Employee Details UI Flow
        • Employee Activation-Deactivation UI Flow
        • Search Employee By Multiple Criteria UI Flow
        • HRMS Service
        • Employees Count UI Flow
        • Auto Escalation UI Flow
      • Events And Notifications
        • User Events Services
        • Citizen Flow
        • Employee Flow - Events
        • Employee Flow - Messages
      • Citizen Engagement Module
        • Document Uploader Service
        • Documents: Implementation Details
      • Common UI Docs
        • User Profile
          • Citizen Profile
          • Employee Profile
          • Employee City Change
        • Update Number & Invalid Number
        • Troubleshoot Using Browser Network Tab
      • Birth & Death
        • Birth & Death User Manual
          • Birth & Death Citizen User Manual
          • Birth & Death Employee User Manual
        • Birth & Death Service Configuration
          • State DSS - Birth and Death
          • Birth & Death Service Setup
      • National Urban Real-Time Dashboard
        • National Urban Dashboard User Manual
        • National Urban Dashboard Functional Specifications
        • National Dashboard Service Configuration
          • National Dashboard - UI Technical Doc
          • DSS UI Flow
          • W&S - National Dashboard
          • Property Tax - National Dashboard
          • Fire NOC - State DSS
          • National Dashboard - Overview
          • National Urban Realtime Dashboard
          • Trade License - National Dashboard
          • OBPS - National Dashboard
          • PGR - National Dashboard
          • National Dashboard Ingest Service
          • National Dashboard API Performance Testing and Benchmark
          • National Dashboard: Index Creation Steps
          • mCollect National Dashboard
      • Product FAQs
    • Training Resources
      • Training Videos
      • Technical Enablement Areas
      • eDCR Training Plan
    • Partner Support
      • eGov Enablement Support for DIGIT
      • Security Guidelines Handbook
      • Troubleshooting Guide
  • 🛃Ecosystem
    • DIGIT Pre-Sales Toolkit
      • Infra Calculator
      • Resource Calculator
      • DIGIT Services
    • Access DIGIT
    • Urban Docs Guide
    • Roadmap
    • Open Events
    • Source Code
Powered by GitBook

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

On this page
  • Overview
  • Pre-requisites
  • Key Functionalities
  • Configuration Details
  • Deployment Details
  • Integration
  • Integration Scope
  • Integration Benefits
  • Steps to Integration

Was this helpful?

Edit on GitHub
Export as PDF
  1. Platform
  2. Configuration
  3. Configuring Services

Persister Service

Overview

Persister service provides a framework to persist data in a transactional fashion with low latency based on a config file. Removes repetitive and time-consuming persistence code from other services.

Pre-requisites

Before you proceed with the documentation, make sure the following pre-requisites are met -

  • Prior Knowledge of Java/J2EE.

  • Prior Knowledge of SpringBoot.

  • Prior Knowledge of PostgresSQL.

  • Prior Knowledge of JSONQuery in Postgres. (Similar to PostgresSQL with a few aggregate functions.).

  • Kafka server is up and running.

Key Functionalities

  • Persist data asynchronously using Kafka providing very low latency

  • Data is persisted in batch

  • All operations are transactional

  • Values in prepared statement placeholder are fetched using JsonPath

  • Easy reference to parent object using ‘{x}’ in jsonPath which substitutes the value of the variable x in the JsonPath with value of x for the child object.(explained in detail below in doc)

  • Supported data types ARRAY("ARRAY"), STRING("STRING"), INT("INT"),DOUBLE("DOUBLE"), FLOAT("FLOAT"), DATE("DATE"), LONG("LONG"),BOOLEAN("BOOLEAN"),JSONB("JSONB")

Configuration Details

Persister uses the configuration file to persist data. The key variables are described below:

  • serviceName: Name of the service to which this configuration belongs.

  • description: Description of the service.

  • version: the version of the configuration.

  • fromTopic: The Kafka topic from which data is fetched

  • queryMaps: Contains the list of queries to be executed for the given data.

  • query: The query to be executed in form of a prepared statement:

    • basePath: base of json object from which data is extracted

    • jsonMaps: Contains the list of jsonPaths for the values in placeholders.

    • jsonPath: The jsonPath to fetch the variable value.

serviceMaps:
 serviceName: student-management-service
 mappings:
 - version: 1.0
   description: Persists student details in studentinfo table
   fromTopic: save-student-info
   isTransaction: true
   queryMaps:
       - query: INSERT INTO studentinfo( id, name, age, marks) VALUES (?, ?, ?, ?);
         basePath: Students.*
         jsonMaps:
          - jsonPath: $.Students.*.id

          - jsonPath: $.Students.*.name

          - jsonPath: $.Students.*.age

          - jsonPath: $.Students.*.marks

Bulk Persister

To persist large quantity of data bulk setting in persister can be used. It is mainly used when we migrate data from one system to another. The bulk persister has the following two settings:

Variable Name

Default Value

Description

persister.bulk.enabled

false

Switch to turn on or off the bulk kafka consumer

persister.batch.size

100

The batch size for bulk update

Any Kafka topic containing data that has to be bulk persisted should have '-batch' appended at the end of topic name example: save-pt-assessment-batch.

Persister Config Versioning

  • Every incoming request [via kafka] is expected to have a version attribute set, [jsonpath, $.RequestInfo.ver] if versioning is to be applied.

  • If the request version is absent or invalid [not semver] in the incoming request, then a default version is defined by the following property in the application.propertiesdefault.version=1.0.0 is used.

  • The request version is then matched against the loaded persister configs and applied appropriately.

Deployment Details

  1. Write configuration as per the requirement. Refer to the example given earlier.

  2. In the environment file, mention the file path of configuration under the variable egov.persist.yml.repo.path while mentioning the file path we have to add file:///work-dir/ as a prefix. for example: egov.persist.yml.repo.path = file:///work-dir/configs/egov-persister/abc-persister.yml. If there are multiple file separate them with comma (,).

  3. Deploy the latest version of egov-persister service and push data on Kafka topic specified in config to persist it in DB.

Integration

Integration Scope

The persister configuration can be used by any module to store records in a particular table of the database.

Integration Benefits

  • Insert/Update Incoming Kafka messages to Database.

  • Add Modify Kafka message before putting it into the database.

  • Persist data asynchronously.

  • Data is persisted in batch.

Steps to Integration

  1. Write configuration as per your requirement. The structure of the config file is explained above in the same document.

  2. Check-in the config file to a remote location preferably Github.

  3. Provide the absolute path of the checked-in file to DevOps, to add it to the file-read path of egov-persister. The file will be added to egov-persister's environment manifest file for it to be read on start-up of the application.

  4. Run the egov-persister app and push data on Kafka topic specified in config to persist it in DB

PreviouseGov OTP ServiceNextEncryption Service

Last updated 3 years ago

Was this helpful?

Each persister config has a version attribute which signifies the service version, this version can contain custom DSL; defined here,

All content on this page by is licensed under a .

GitHub - zafarkhaja/jsemver: Java implementation of the SemVer Specification
​
eGov Foundation
Creative Commons Attribution 4.0 International License
Creative Commons License