Overview

The main objective of the billing module is to generate the bill for all revenue-based business services. To serve the bill, the Billing-Service requires demand. Demands will be prepared by the revenue modules and stored by billing based on which it generates the Bill.

Pre-requisites

• Prior Knowledge of Java/J2EE.

• Prior Knowledge of Spring Boot.

• Prior Knowledge of KAFKA

• Prior Knowledge of REST APIs and related concepts like path parameters, headers, JSON, etc.

• Prior knowledge of demand-based systems.

• The following services should be up and running:

  • user

  • MDMS

  • Id-Gen

  • URL-Shortening

  • notification-sms

Key Functionalities

• eGov billing service creates and maintains demands.

• Generates bills based on demands.

• Updates the demands from payment when the collection service takes a payment.

Deployment Details

• Deploy the latest image of the billing service available.

Configuration Details

In the MDMS data configuration, the following master data is needed for the functionality of the billing.

MDMS

Business Service JSON

{
  "tenantId": "pb",
  "moduleName": "BillingService",
  "BusinessService": [
    {
      "businessService": "PropertyTax",
      "code": "PT",
      "isBillAmendmentEnabled":"true",
      "collectionModesNotAllowed": [
        "DD","OFFLINE_NEFT","OFFLINE_RTGS","POSTAL_ORDER"
      ],
      "partPaymentAllowed": true,
      "minAmountPayable":100,
      "isAdvanceAllowed": false,
      "demandUpdateTime": 86400000,
      "isVoucherCreationEnabled": true,
      "billGineiURL" : "egov-searcher/bill-genie/billswithaddranduser/_get"
    },
    {
      "businessService": "WaterCharges",
      "code": "WC",
      "isBillAmendmentEnabled":"true",
      "collectionModesNotAllowed": [
        "DD","OFFLINE_NEFT","OFFLINE_RTGS","POSTAL_ORDER"
      ],
      "partPaymentAllowed": false,
      "isAdvanceAllowed": true,
      "demandUpdateTime": 86400000,
      "isVoucherCreationEnabled": false
    },
    {
      "businessService": "TradeLicense",
      "code": "TL",
      "collectionModesNotAllowed": [
        "DD","OFFLINE_NEFT","OFFLINE_RTGS","POSTAL_ORDER"
      ],
      "partPaymentAllowed": false,
      "isAdvanceAllowed": false,
      "demandUpdateTime": 604800000,
      "isVoucherCreationEnabled": true
    }
  ]
}

TAX-Head JSON

{
  "tenantId": "pb",
  "moduleName": "BillingService",
  "TaxHeadMaster": [
    {
      "category": "ADVANCE_COLLECTION",
      "service": "PT",
      "name": "Pt advance carry forward",
      "code": "PT_ADVANCE_CARRYFORWARD",
      "isDebit": true,
      "isActualDemand": false,
      "order": "0",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "Pt owner exemption",
      "code": "PT_OWNER_EXEMPTION",
      "isDebit": true,
      "isActualDemand": true,
      "order": "5",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "Pt time rebate",
      "code": "PT_TIME_REBATE",
      "isDebit": true,
      "isActualDemand": true,
      "order": "2",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "Pt unit usage excemption",
      "code": "PT_UNIT_USAGE_EXEMPTION",
      "isDebit": true,
      "isActualDemand": true,
      "order": "6",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "Pt adhoc penalty",
      "code": "PT_ADHOC_PENALTY",
      "isDebit": false,
      "isActualDemand": false,
      "order": "1",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "propertytax",
      "code": "PT_TAX",
      "isDebit": false,
      "isActualDemand": true,
      "order": "0",
      "isRequired": false
    },
    {
      "category": "TAX",
      "service": "PT",
      "name": "Pt fire cess",
      "code": "PT_FIRE_CESS",
      "isDebit": false,
      "isActualDemand": true,
      "order": "7",
      "isRequired": false
    }
  ]
}

Tax-Period JSON

{
  "tenantId": "pb",
  "moduleName": "BillingService",
  "TaxPeriod": [
    {
      "fromDate": 1554076799000,
      "toDate": 1585679399000,
      "periodCycle": "ANNUAL",
      "service": "PT",
      "code": "PTAN2019",
      "financialYear": "2019-20"
    },
    {
      "fromDate": 1522540800000,
      "toDate": 1554076799000,
      "periodCycle": "ANNUAL",
      "service": "PT",
      "code": "PTAN2018",
      "financialYear": "2018-19"
    },
    {
      "fromDate": 1491004800000,
      "toDate": 1522540798000,
      "periodCycle": "ANNUAL",
      "service": "PT",
      "code": "PTAN2017",
      "financialYear": "2017-18"
    },
    {
      "fromDate": 1459468800000,
      "toDate": 1491004799000,
      "periodCycle": "ANNUAL",
      "service": "PT",
      "code": "PTAN2016",
      "financialYear": "2016-17"
    },
    {
      "fromDate": 1522540800000,
      "toDate": 1554076799000,
      "periodCycle": "ANNUAL",
      "service": "TL",
      "code": "TLAN2018",
      "financialYear": "2018-19"
    }
  ]
}

Integration Details

Integration Scope

Billing service can be integrated with any organization or system that wants a demand-based payment system.

Integration Benefits

• Easy to create and simple process of generating bills from demands

• The amalgamation of bills period-wise for a single entity like PT or Water connection.

• Amendment of bills in case of legal requirements.

Steps to Integration

1. Customers can create a demand using the /demand/_create

2. Organizations or Systems can search the demand using /demand/_searchendpoint

3. Once the demand is raised the system can call /demand/_update endpoint to update the demand as per need.

4. Bills can be generated using, which is a self-managing API that generates a new bill only when the old one expires /bill/_fetchbill.

5. Bills can be searched using /bill/_search.

6. Amendment facility can be used in case of a legal issue to add values to existing demands using /amendment/_create and /amendment/_update can be used to cancel the created ones or update workflow if configured.

Interaction Diagram

Interaction Diagram V1.1:

Reference Docs

Doc Links

API List

Apportioning FAQs

What is apportioning?

Adjusting the receivable amount with the individual tax head.

Types of apportioning V1.1

Default order-based apportioning(Based on apportioning order adjust the received amount with each tax head).V1.1

Types of apportioning V1.2: (TBD)

• Proportionate-based apportioning (Adjust total receivable with all the tax heads equally)

• Order & percentage-based apportioning (Adjust total receivable based on order and the percentage which is defined for each tax head).

Principle of apportioning

The basic principle of apportioning holds that if the full amount is paid for any bill then each individual tax head should get nullified with their corresponding adjusted amount.

Example: Case 1: When there are no arrears all tax heads belong to their current purpose.

Example: given below

Case 2: Apportioning with two years of arrear: Example: The apportioning details for the financial year 2014-15 are given below.

The table below illustrates the demand structure generated in case there are no payments for the specified financial year (2015-16).

Overview

The Collection service is to serve as a revenue collection platform for all the billing systems through cash, cheque, demand draft, and swipe machines. It enables payment for all services provided by the eGov platform at a single point for the citizen and counter collection in municipal alike.

Pre-requisites

• Prior knowledge of Java/J2EE

• Prior knowledge of SpringBoot

• Prior knowledge of REST APIs and related concepts like path parameters, headers, JSON, etc

• Prior knowledge of Kafka and related concepts like Producer, Consumer, Topic, etc.

• The following services should be up and running:

  • egov-localization

  • egov-mdms

  • egov-idgen

  • egov-url-shortening

  • billing-service

Key Functionalities

• Allows citizens to create a payment.

• Allows employees to create the payment for the citizen indirectly.

• provides facilities to capture partial and advanced payments based on configs.

• allows payment cancellation to help with scenarios of bad checks and other failed payment scenarios.

• Integrates with billing service for demand back-update of payment.

Deployment Details

• deploy the latest version of the collection-services docker builds.

Configuration Details

The MDMS data configuration uses the same data updated by Billing-Service

Following are the properties in the application.properties

Integration Details

Integration Scope

Collection service can be integrated with any organization or system that wants a payment system to keep track of its payments. Organizations can customize part of the application or its functionality based on their requirements.

Integration Benefits

• Easy payments and tracking of payments.

• Configurable functionalities according to client requirements.

Steps to Integration

• Customers can create a payment using the /payments/_create.

• Actors on the system can keep track of payments using /payments/_searchendpoint.

• Once the payment is done but it encounters a technical issue outside of the system then it can be cancelled with /payments/_workflow.

• For employees to access the payments API the respective module name should be appended after the payment API path - /payments/PT/_workflow - here PT refers to the property module.

IFSC Code Migration - Collection Service

• Port forward the collection service to the current environment where the IFSC CODE bank details data is to be migrated. Sample command - 1kubectl port-forward collection-services-76b775f976-xcbt2 8055:8080 -n egov

• Import postman collection from API list which refers as /preexistpayments/_update and runs with the same localhost to where we port forwarded using the above command.

• Expected result. In the EGCL_PAYMET table where IFSCODE data is present for those records, EGCL_PAYMET.ADDITIONALDETAILS bankdetails will be updated.

Interaction Diagram

Reference Docs

Doc Links

API List

Overview

The Collection Service serves as a revenue collection platform for all the billing systems through cash, cheque, demand drafts, or the swipe machine. It enables payment for all services provided by the eGov platform at a single point directly from the citizen or over-the-counter collection within municipalities.

Pre-requisites

• Prior Knowledge of Java/J2EE

• Prior Knowledge of SpringBoot

• Prior Knowledge of REST APIs and related concepts like path parameters, headers, JSON, etc

• Prior Knowledge of Kafka and related concepts like Producer, Consumer, Topic, etc.

• The following services must be up and running:

  • egov-localization

  • egov-mdms

  • egov-idgen

  • egov-url-shortening

  • billing-service

Key Functionalities

• Allows citizens to create a payment.

• Allows employees to create the payment for the citizen indirectly.

• Provides facilities to capture partial and advanced payments based on configs.

• Allows payment cancellation to help with scenarios of bad checks and other failed payment scenarios.

• Integrates with billing service for demand back-update of payment.

Deployment Details

• deploy the latest version of the collection-services docker builds.

Configuration Details

The MDMS data configuration uses the same data updated by the Billing-Service.

The table below lists the application properties.

Integration

Integration Scope

Collection service can be integrated with any organization or system that requires a payment system to keep track of its payments. Organizations can customize part of the application or its functionalities based on their requirements.

Integration Benefits

• Easy payments and tracking of payments.

• Configurable functionalities according to client requirement

Steps to Integration

1. Customers can create a payment using the /payments/_create

2. Actors on the system can keep track of payments using /payments/_searchendpoint

3. Once the payment is done and it encounters a technical issue that is beyond the system - the payment can be cancelled with /payments/_workflow

4. For employees to access the payments API the respective module name should be appended to the payment API path - /payments/PT/_workflow. Here PT refers to the property module.

Interaction Diagram

Reference Docs

Doc Links

API List

Overview

The consumer sometimes needs additional amounts (Amendments) added to their bill due to reasons from outside of the system. The addition of amounts happens with respect to the consumer code of the entity in the product(PT, WS, etc..,), any unpaid demand in the system is a candidate for amendments.

Pre-requisites

• Prior knowledge of billing-service in the DIGIT framework.

Key Functionalities

Amendment mainly works with two types of functionality as follows:

• Amendment

• Demand

Bill Amendment provides a separate flow that triggers the workflow for validating the process of adding additional amounts to existing demands. This validation was earlier available only to the respective modules. An amendment is allowed only when there is a need to add or reduce the amount from the existing bill belonging to an entity. The reasons for such cases could be:

1. Court case settlement

2. One time waiver

3. Write-offs

4. DCB correction (old demands in paid status)

5. Property tax remission

Criteria:

Below are certain prerequisites to creating an amendment,

1. presence of demand in the billing system

2. any one of the reasons listed above

3. valid document proof for the reason

4. there is no other amendment already in the workflow

Procedure:

The process of adding an amendment in specific scenarios is given below.

There are two scenarios on how an amendment is completed based on the paid status of the existing demands in the system.

1. when demand is unpaid/partially paid

1. Create a demand (Or an existing demand can be used) with demand detail → DD1.

2. Do not pay the bill or make a partial payment.

3. Create an amendment for the same consumer code (with demand detail → DD2).

4. Approve the amendment - the response should return an amendment with the status CONSUMED.

5. Search the demand or fetch the bill for the consumer code. The demand/bill should contain demand details of the demand and amendment together with DD1 and DD2 in the same demand/bill.

2. when demand is completely paid

1. Create demand and make complete payment or choose a consumer code which is fully paid.

2. Create amendment (with demand detail → DD1).

3. Approve amendment - the response should be APPROVED.

4. Create new demand for the consumer code (with demand detail → DD3). The demand response should contain two demand details DD1 and DD2 saved to the demand.

5. The amendment search returns CONSUMED status after the demand is created.

IMPACT: Does not impact any other functionality other than adding demand details to demands on APPROVAL.

IMPACTED BY: Existence of demands in the system.

Configuration Details

WORKFLOW CONFIG:

Integration Details

Integration Scope

Amendment integration helps organizations add additional value to the demand without any change in the system.

Integration Benefits

• Easy to create and simple process of updating demands

• Helps ease changes into the system which are not part of normal functionality - Amendment of bills in case of legal requirements.

Steps to Integration

This is integrated into the billing system by default.

• The amendment facility can be used in case of a legal issue to add values to existing demands using /amendment/_create. The parameter /amendment/_update is used to cancel the created updates or update configured workflows.

Doc Links

API Definition

API List

This section provides technical details about business service setup, configuration, deployment, and API integration.

Overview

This release for DSS focuses on improving user experience and the ability given to the user to get deeper insights using drill-through and comparison indicators in tables.

The release includes the following features:

• Breadcrumbs for better navigation

• Drill through options in tables and charts

• Comparison indicators in Table

Breadcrumbs For Navigation

In addition to the left navigation panel, the addition of breadcrumbs is also useful to provide a better sense of the current page insight. It is also very much helpful for mobile navigation. The user can navigate using the breadcrumbs by clicking on the required parent menu.

Technical Implementation Details

It Works based on the Current Route URL and previous Route URL

File Details - https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/Breadcrumbs.js

Drill through options in tables and charts

The ability provided in DSS to configure the drill through for required options in tables as well as charts. The drill through options is useful in configuring the required hierarchy of data set. This helps users to go up to 'N' levels to get deeper insights

Technical Implementation Details:

Drill down/drill through in tables, is based on the drillDownChartId and filter.

Here chart id is used for the subsequent call to fetch the next table along with the applied/selected filters.

File Details - https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/components/Charts/TableChart.js

Drill throughs in piecharts:

It is similar to the drill-down in tables. Here drill through in piecharts are based on the drillDownChartId field in the parent piechart.

File Details - https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/components/Charts/DonutChart.js

Comparison Indicators in Tables

Providing better insights about the metric performances of different dimensions, a comparison indicator is required inside data tables comparing usually with a different time range (last year/last month) and what is percentage change with time.

Technical Implementation Details:

Comparison with the previous year's data in every table data uses the same request object by changing the time range to the previous year/month/week.

File Details - https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/components/Charts/TableChart.js

The following method along with parameters is used to fetch the previous year's data.

getLastYearRequest(calledFrom, visualcode, active, filterList) 

After receiving last year's data it is compared with the current year's data. The comparison is shown as insight data. The comparison logic is present in uiTable.js -

https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/components/common/UiTable/UiTable.js

TimeFilter

The current time component is not very intuitive and user-friendly. So a new library react-date range is used to enhance the time filter.

File Details - https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/components/common/DateRange/index.js

Event Duration Graphs

Ability to generate graphs showcasing time spent between multiple events like average turnaround time, complaint assigning time, etc.

A DSS_EVENT_DURATION_GRAPH is added in the PGR config

{

    "id": 412,

    "name": "DSS_EVENT_DURATION_GRAPH",

    "dimensions": {

        "height": 350,

        "width": 7

    },

    "vizType": "chart",

    "isCollapsible": false,

    "label": "",

    "charts": [

        {

            "id": "eventDurationGraph",

            "name": "Monthly",

            "code": "",

            "chartType": "line",

            "filter": "",

            "headers": []

        }

    ]

}

API Call Role Action Mapping

Overview

A decision support system (DSS) is a composite tool that collects, organizes and analyzes business data to facilitate quality decision-making for management, operations and planning. A well-designed DSS aids decision makers in compiling a variety of data from many sources: raw data, documents, personal knowledge from employees, management, executives and business models. DSS analysis helps organizations identify and solve problems, and make decisions.

Code Git Repos: https://github.com/egovernments/frontend/tree/master/web/dss-dashboard

Type Of Users

1. State-Level Admin

2. Commissioner

3. Domain-Level Employee

Dashboard List

There are three types of dashboards -

1. Home page (refer figure 1)

2. Overview page (refer figure 2)

3. Module level dashboard (refer figure 3)

1. Home page

The home page contains multiple cards, each card is clickable.

There are two types of cards, i.e, the overview card and the module-level card.

The overview and the module level cards are differentiated by vizType,

1. Overview card: Clicking on the overview card navigates to the overview page. vizType for overview is a collection.

2. Module Level card: Clicking on the module level card navigates to the module level dashboard. vizType is a module (i.e Property Tax, Trade License etc).

Request Payload for dashboardConfig

auth-token: authenticate the request and it fetches from a local storage key called “Employee.token”

DashboardConfig API Response

roleName: the type of user.

Visualisations: The key contains all configurations for displaying the visualisation like rows with charts etc please refer to figure 1.3.

In Figure 1.3, vizType key will define the module UI like

Collection chart & module chart refer figure 1

2. Overview Dashboard

3. Module Level Dashboard

Visualisation List

In dashboardConfig response, the visualisation key contains all rows & charts details (refer figure 1.3). Each row contains the visual details like name, vizType, noUnit, isCollapsible, charts etc (refer figure 1.3).

1. name - name of visualisation

2. vizType - type of visualisation like COLLECTION, MODULE, METRIC-COLLECTION, PERFORMING-METRIC, CHART

   • COLLECTION - The home page, contains the collection data (refer figure 1).

   • MODULE - The home page, contains the module-level data (refer figure 1).

   • METRIC-COLLECTION - In Overview/Module Level Page, contains the collection data (refer figure 2.1).

   • PERFORMING-METRIC -In Overview/Module Level Page, contains the top/bottom performing data (refer figure 2.2).

   • CHART - In Overview/Module Level Page, contains the below visualisations (refer figure 2.3 to figure 2.7).

     • PIE CHART (refer figure 2.3)

     • LINE CHART (refer figure 2.4)

     • BAR CHART (refer figure 2.5)

     • HORIZONTAL BAR CHART (refer figure 2.6)

     • TABLE CHART (refer figure 2.7)

Visualisations

ULB Dashboard

ULB dashboard contain different filters, i.e ULBs and Wards/Blocks. The data to the filters are loaded from MDMS API below - https://dev.digit.org/egov-mdms-service/v1/_search

1. Each ULB dashboard, overview dashboard and module-level pages contain different filters and are identified by roleName in configs API.

2. The Wards/Blocks filter is a dependable filter, which gets loaded on ULB selection.

3. In the ULB dashboard, the on-page ULB filter is applied across all the charts and for the performance chart, the default ULB filter is not applied.

4. Overview and all module level pages has a ULB dashboard.

GLOBAL Filters (refer to figure 2.8)

Filters are loaded from the MDMS API - https://dev.digit.org/egov-mdms-service/v1/_search. Filters are loaded on the basis of roleName.

Admin role: On the Module level page, Date, DDR and ULB filter are loaded.

On the Overview level page, Date, DDR, ULB and Service filter are loaded.

Commissioner role: On the Module level page, Date, ULB and Wards/Blocks filters are loaded.

On the Overview page, Date, ULB and Service filters are loaded.

Denomination filter: The Denomination filter has three options to display the amount and number in a particular format.

1. Crore

2. Lack

3. Unit

The denomination filter is not applied to the percentage and text (refer to figure 2.10). The type of data is identified by a symbol in the plots of charts API.

Custom Date Filter

If duration < 15 days, it displays data day-wise

If duration <= 30 days, it displays data week-wise

If duration >30, it displays data month-wise

Tabs

Currently, the dashboard contains two types of tabs -

1. Revenue (refer figure: 4.1)

2. Service (refer figure: 4.1)

Tabs are identified by name in visualisations of config API.

Table Chart with drill-down

Table chart visualisations have normal material UI data table features like search, sort etc.

In table response, if filter key & drillDownChartId contain any value users can drill-down the table.

Cards

1. Each card header is localised and has an info icon with a tooltip option that displays the header and can display a description.

2. The number of cards in a row and in a page is driven by the backend. The backend provides the row number to each card where it should be displayed.

3. Card contains option icon that enables users to either download images and or share images.

4. Image download and share user id from vizArray in order to differentiate each card in a page.

Download and Share (refer to figure 2.9)

Download offers two options - to download data as an image or a PDF.

Share: Share creates the Image/PDF and uploads it S3 using below API and returns file id - https://mseva-uat.lgpunjab.gov.in/filestore/v1/files

The file Id is fetched using the API - https://mseva-uat.lgpunjab.gov.in/filestore/v1/files/url

Each S3 image is shortened using the API - https://mseva-uat.lgpunjab.gov.in/egov-url-shortening/shortener

Configurations

Github link for config: https://github.com/egovernments/frontend/blob/master/web/dss-dashboard/src/config/configs.js

BASE URL: End point of REST API for dashboard.

FILE Upload: End point of REST API for file upload.

FETCH FILE: End point of REST API for file fetch.

MDMS: End point of REST API for fetch MDMS Data.

SHORTEN URL: End point of REST API for Shorten URL, which is used for share via Email / What's app.

CHART COLOR CODE: Color code object for all charts.

MODULE LEVEL: for global filters, which contains services name & filter key.

SERVICES: for global filter, service filter.

Upload Localisation Keys

code: pre-defined key for back-end.

message: message contains the value for the key.

module: rainmaker-dss

locale: contains locale data

for more details eGov team to be documented

Module name: rainmaker-dss

NPM Module Used - https://github.com/egovernments/DIGIT-OSS/blob/master/frontend/mono-ui/web/dss-dashboard/package.json

Steps to setup DSS in Local

Step 1: Run as independent, switch to dss-dashboard folder

Step 2: Get the below details from the environment website and update the localstorage in the browser.

Employee.tenant-id Employee.user-info Employee.token Employee.module Employee.locale localization_en_IN locale

Step 3: Run Yarn install and yarn start to start working on dss in local setup.

DSS Features Enhancements V2: DSS Features Enhancements V2 Technical Document for UI

Overview

DSS has two sides to it. One is the process in which the data is pooled to ElasticSearch and the other is the way it is fetched, aggregated, computed, transformed and sent across. DSS must be configurable since the entire process involves playing around with a variety of data sets. This ensures easy configuration of data sets in new scenarios.

This document explains the steps on how to define the configurations for both sides of DSS Analytics and Ingest Pipeline Services.

Commonly Used Terms

Ingest: Micro Service which runs as a pipeline and manages to validate, transform and enrich the incoming data and pushes the same to ElasticSearch Index

Analytics: Micro Service which is responsible for building, fetching, aggregating and computing the Data on ElasticSearch to a consumable Data Response. Which shall be later used for visualizations and graphical representations.

JOLT: JSON to JSON transformation library written in Java where the "specification" for the transform is itself a JSON document

Modules / Domain Level: These are the Services in this context. Each of the services, such as Property Tax, Trade License, Water and Sewerages are considered as Modules / Domains

Chart: Each individual graphical representation is considered as a Chart in specific. For example, a Metric of Total Collection is considered as a Chart.

Visualization: Group of different Charts is considered as a Visualization. For example, the group of Total Collection, Target Collection and Target Achieved is considered as a Metric Collection of Charts and thus it becomes a Visualization.

Configuration Details

Discussed below are the ingest pipeline configuration details -

1. Topic Context Configurations

2. Validator Schema

3. JOLT Transformation Schema

4. Enrichment Domain Configuration

5. JOLT Domain Transformation Schema

Topic Context Configurations

Topic context configuration is an outline to define which data is received on which Kafka Topic.

Indexer Service and many other services are sending out data on different Kafka Topics. If the Ingest Service receives the data and passes it through the pipeline, the context and the version of the data received have to be set. This configuration is used to identify which Kafka topic consumed the data and the mapping details.

Click here for the full configuration

Validator Schema

Validator schema is a configuration schema library from Everit. By passing the data against this schema, it ensures whether the data abides by the rules and requirements of the schema that has been defined.

Click here for an example configuration

JOLT Transformation Schema

JOLT is a JSON to JSON transformation library. In order to change the structure of the data and transform it in a generic way, JOLT has been used.

While the transformation schemas are written for each data context, the data is transformed against the schema to obtain transformed data.

Follow the slide deck for JOLT Transformations

Click here for an example configuration

Enrichment Domain Configuration

This configuration defines and directs the enrichment process that the data goes through.

For example, if the data which is incoming is belonging to a collection module data, then the collection domain config is picked. And based on the specified business type the right config is picked.

In order to enhance the data of collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set.

Click here for an example configuration

JOLT Domain Transformation Schema

As a part of enhancement, once the domain level object is obtained, we might not need the complete document as is in the end data product.

Only those parameters which should be or can be used for aggregation and representation are to be held and others are to be discarded.

In order to do that, we make use of JOLT again and write schemas to keep the required ones and discard the unwanted ones.

The above configuration is used to transform the data response in the enrichment layer.

Click here for an example configuration

Use case:- JOLT Transformation Schema for collection V2

JOLT transformation schema for payment-v1 has been taken as a use case to explain the context collection and context version v2. The payment records are processed/transformed with the schema. The schema supports splitting the billing records into an independent new record. So if there are 2 bill items in the collection/payment incoming data then this results in 2 collection records in turn.

Click here for an example configuration

Here: $i, the variable value that gets incremented for the number of records of paymentDetails

$j, the variable value that gets incremented for the number of records of bill details.

Note: For Kafka connect to work, Ingest pipeline application properties or in environments direct push must be disabled.

es.push.direct=false

Analytics Configurations

Below is the list of configurations

1. Chart API Configuration

2. Master Dashboard Configuration

3. Role Dashboard Mappings Configuration

Chart API Configuration

Each Visualization has its own properties. Each Visualization comes from different data sources (Sometimes it is a combination of different data sources)

In order to configure each visualization and its properties, we have Chart API Configuration Document.

In this, Visualization Code, which happens to be the key, will be having its properties configured as a part of the configuration and are easily changeable.

Click here for an example configuration

Master Dashboard Configuration

Master dashboard configuration defines the dashboards that are to be painted on the screen.

It includes all the visualizations, their groups, the charts and even their dimensions in terms of height and width.

Click here for an example configuration

Role Dashboard Mappings Configuration

Role dashboard mapping ensures that each role is mapped against the dashboards that they are authorized to see.

Click here for an example configuration

Adding Configurations

Adding Roles and Dashboards

To add a new role, modify the RoleDashboardMappingsConf.json (roles node) configuration file as given below.

Below as in Figure 9 is a sample to add a new role object, a new dashboard object.

To add a new dashboard, modify the MasterDashboardConfig.json (dashboards node) as shown below in Figure 10.

Adding Visualizations

To add new visualisations, modify the MasterDashboardConfig.json (vizArray node) as shown in Figure 11.

Adding Charts

To add a new chart, chartApiConf.json has to be modified as shown below. A new chartid has to be added with the chart node object.

Metric chart Sample as shown in Figure 12.

Pie chart Sample as shown in Figure 13.

Line chart Sample as shown in Figure 14.

Table chart Sample: This chart is of 2 types - table and xtable.table (as shown in Figure 15.) Type allows the addition of aggregated fields as available in the query keys. To extract the values based on the key, aggegationPaths have to be added along with their data type as in pathDataTypeMapping.

xtable (as shown in Figure 16) type allows the addition of multiple computed fields with the aggregated fields added dynamically.

To add multiple computed columns, define the following params within computedFields []

• actionName - (IComputedField<T> interface),

• fields - [] names as existing in the query key,

• newField - name to appear for the computation

https://github.com/egovernments/configs/blob/master/egov-dss-dashboards/dashboard-analytics/ChartApiConfig.json for the full configuration in detail.

Steps to create charts and visualise are:

• Create/Add a chart in chartApiConf.json

• Add a visualization for the existing dashboard in MasterDashboardConfig.json as defined above.

• Or in order to create/add a new dashboard create the dashboard in MasterDashboardConfig.json and create a role in RoleDashboardConfig.json

Configuration Changes For DrillThroughs:

Example - drill through in ward table in the property tax dashboard.

wardDrillDown is the visualization code for PT Drill Down. The 'kind' attribute shows the type of visualization code. Apart from two things all the attributes are common.

Example - Drill through in the ComplaintList table in the PGR Dashboard.

complaintDrillDown is the visualization code for PGR Drill Down.

The above complaintDrillDown visualization code is called in the drill chart parameter.

Overview

Apportion service is used to apportion the amount paid against a bill among the different tax heads based on the implemented algorithm. The default algorithm uses the order of the tax head to apportion the tax head with the lowest order apportioned off first and the highest order tax head apportioned last.

Pre-requisites

Before you proceed with the documentation, make sure the following pre-requisites are met -

• Java 8

• Kafka server is up and running

• egov-persister service is running and has apportioned persister config path added to it

• PSQL server is running and a database is created to store apportion audit data

Key Functionalities

• Apportion payment in tax heads of bill

• Apportion advance amount in tax heads of demand during demand creation

Deployment Details

1. Deploy the latest version of egov-apportion-service

2. Add apportion persister yaml path in persister configuration

Configuration Details

There is no separate configuration required. The TaxHead master that is configured in the billing service is only used

Integration Details

Integration Scope

Any payment service which wants to divide the paid amount into different tax head buckets can integrate with apportion service.

Integration Benefits

• Apportions amount in tax heads

Steps to Integration

1. To integrate, the host of egov-apportion-service should be overwritten in the helm chart

2. /apportion-service/v2/bill/_apportion should be called to apportion the bill

3. /apportion-service/v2/demand/_apportion should be called to apportion the advance amount in demands

Interaction Diagram

Reference Docs

Doc Links

API List

(Note: All the APIs are in the same postman collection therefore the same link is added in each row)

Overview

DIGIT is India's largest open-source platform for Urban Governance. It provides API-based access to government functions enabling the government to provide facilities via integration with relevant service players. This document provides the details of how system integrators enable bill collection facilities to customers using DIGIT as the governance platform. It outlines the integration approach with Billing and Collections services to enable fetching bill dues to citizens and recording their payments into the system.

Integration Approach

DIGIT is completely API driven and allows for data exchange with disparate systems using REST API calls. Most functional APIs are protected resources that can be accessed after proper authentication with the platform. The platform also checks for the right level of access for given credentials. A bill collection flow -

1. Authenticate with DIGIT

2. Get citizen bill using a service-specific query

3. Record the payment details against the bill

4. Optional - Get payment API to fetch the receipt details

The in-field team of the system integrator makes the calls to the integrator's own system (or a standard system like BBPS). Integration with DIGIT follows a server-to-server approach where the backend system of the integrator makes these calls to the DIGIT platform as per requirement. The diagram below depicts the high-level flow of calls between on-field devices like PoS to the integrator backend (Integrator System) and from the backend of the DIGIT integrator to DIGIT (DIGIT Platform).

Integration Details

DIGIT uses Swagger 2.0 as its API standard and all its APIs are documented in Swagger. Wherever needed this document provides a link to our API documentation online. An example of typical request/response snippets necessary for integration is provided below in the respective sections.

DIGIT is a multi-tenanted system - hence all APIs in DIGIT except tenantid are passed either in the query param or RequestBody (Please refer to detailed API documentation as indicated in sections below). The tenantid represents the modular operating unit for the operation of an API, e.g. in a municipal governance use case. A tenantid represents one ULB. Your platform contact will help you access the configured list for your use case.

Authentication API also expects tenantid (your platform contact will help you identify the one to use). Based on the role as an integrator the OAUTH token in response can be used for unit/ULB level tenants in subsequent API calls (meaning you may not need one authentication per unit/ULB level tenant).

1. Authentication

To ensure data privacy and security, transactional APIs in DIGIT are protected under authentication. System integrators are requested to contact the respective state authority to get the necessary OAUTH tokens required to access the APIs.

Use the API below to generate the access token based on the credentials provided. Given below is an example of the request and response. The OAuth token to be used from the response is highlighted in bold.

Request Snippet

curl -X POST \
    https://egov-micro-dev.egovernments.org/user/oauth/token \
    -H 'Authorization: Basic {Authorization Token}' \
    -H 'Content-Type: application/x-www-form-urlencoded' \
    -H 'Postman-Token: 81c87b4c-f69d-464a-8c9a-990e698391ec' \
    -H 'cache-control: no-cache' \
    -d 'grant_type=password&scope=read&username={username}&password={Password}&tenantId={TenantId}&userType=Employee'

Response Snippet

{
"access_token": "6cf92021-8433-4da3-9a1e-8bc511983b2f",
"token_type": "bearer",
"refresh_token": "1f3097f3-39ab-4fd1-91a9-2ce48ac5b6bc",
"expires_in": 604799,
"scope": "read"
}

2. Fetching Bill

DIGIT allows the integrators to fetch the bills for citizens using the consumer number of the respective service (e.g. Water charges, Property Service, Trade License).

For some services, DIGIT also provides the facility to fetch bills by mobile number.

To support the partial payment use case each bill in the response of the fetch bill API indicates whether it allows partial payment and if yes, the minimum amount to be paid.

To fetch a bill from DIGIT, make sure that the OAuth token is generated as per the Authentication section above. Post that use the following API to fetch the bill -

• https://www.getpostman.com/collections/900d99a85d083fb2d377

• https://raw.githubusercontent.com/egovernments/business-services/master/Docs/billingservice/V-2.0.yml

• Choose Billing Service from the dropdown.

• Go to the Bill section of BillingService.

• Go to the Bill tab.

3. Make Payment

Once the bill is fetched from the DIGIT system, the system integrator is expected to relay it back to the Field Device. The integrator is expected to Initiate and collect the payment based on government preference indicated in the bill (can it be partially paid and if so the minimum amount etc.) and citizen's preference of payment instrument etc.

Once the payment is successfully done in the integrator's system, the integrator is expected to register the payment in DIGIT using the Payment Create API.

DIGIT expects a receipt (result of calling payment API) to be created against the bill number returned in the fetch bill API.

To create a receipt in DIGIT, make sure that the OAuth token is generated as per the Authentication section above. Post that use the following API to create the receipt -

• https://www.getpostman.com/collections/824d6b50b728bccd86d4

• https://raw.githubusercontent.com/egovernments/business-services/master/Docs/collection-services/V-2-0.yml

• Choose Collection Service from the dropdown.

• Go to Payment.

• Go to Make Payment.

Overview

A decision support system (DSS) is a composite tool that collects, organizes and analyzes business data to facilitate quality decision-making for management, operations and planning. A well-designed DSS aids decision-makers in compiling a variety of data from many sources: raw data, documents, personal knowledge from employees, management, executives and business models. DSS analysis helps organizations identify and solve problems, and make decisions.

API Details

The Swagger API for the backend is below

https://github.com/egovernments/business-services/blob/master/Docs/dss-dashboard/DSS%20Analytics%20Dashboard%20YAML%20Spec%201.0.0.yaml

Swagger API for ingest

https://github.com/egovernments/business-services/blob/master/Docs/dss-dashboard/DSS%20Ingest%20YAML%20Spec%201.0.0.yaml

The target upload file template is given below -

__All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

Overview

According to the new collection service, which follows the payment structure for storing information about payments and payment details, it is necessary to migrate the old collection structure into the new payment structure.

Migration Details

In the old collection service, for every transaction, the receipt number is generated on the bill detail level. Since the bill contains multiple bill details each transaction is mapped to multiple receipt numbers. So after payment of a single bill, multiple receipt numbers are generated. The mapping of the transactions to the receipt number changed in the new collection service.

In the new collection service, the receipt number is generated at the bill level. For each bill transaction, one receipt number is generated. So every bill for a consumer code and business service has one receipt number.

The records from tables egcl_receiptheader, egcl_receiptdetails, egcl_instrument, egcl_instrumentheader need to be transferred into tables egcl_payment, egcl_paymentdetail, egcl_bill, egcl_billdetial, egcl_billaccountdetail.

For smooth data transactions, the record from the old receipt is mapped according to the payment structure. The new payment response can be formed with receipt data.

The table below provides the mapping between receipt and payment structure with some remarks.

After the creation of the payment response with receipt data, it is pushed into the Kafka topic “egov.collection.migration-batch”. The persister inserts the payment data into tables - egcl_payment, egcl_paymentdetail, egcl_bill, egcl_billdetial, egcl_billaccountdetail.

Indexer config for the legacy data index and new payments.

persister config -

These need to get promoted before initiating the migration process. Migration happens through an API call, add role-actions based on your requirement. Otherwise, port-forwarding will work.

API Details

Endpoint: /collection-services/payments/_migrate?batchSize=100&offset= Body: { "RequestInfo": { "apiId": "Rainmaker", "action": "", "did": 1, "key": "", "msgId": "20170310130900|en_IN", "ts": 0, "ver": ".01", "authToken": "a6ad2a1b-821c-4688-a70e-4322f6c34e54" }

In case of any failure and restarting migration, take the value of offset and tenantId printed in the logs and resume the migration process.

/collection-services/payments/_migrate?batchSize=100&offset=200&tenantId='pb.tenantId'

Collection-service build:- collection-services-db:9-COLLECTION_MIGRATION-e9701c4

Overview

This specifies the migration steps which are specific to the payment index.

Steps

Step 1: Adding a target index - Add index name dss-payment_v2 as below:

In kibana, dev tools, apply the below command

Step 2: Optional changes required in Ingest application properties

Ingest pipeline application properties contain es.direct.push supposed to be set true for testing.

Step 3: Run migration Api, which migrates the data from the source index to the target index.

In kibana, dev tools verify using below command