DSS Configuration And Setup
Overview
A decision support system (DSS) is a composite tool that collects, organizes, and analyzes business data to facilitate quality decision-making for management, operations, and planning. A well-designed DSS aids decision-makers in compiling a variety of data from many sources: raw data, documents, and personal knowledge from employees, management, executives, and business models. DSS analysis helps organizations identify and solve problems, and make decisions
This document explains the steps on how to define the configurations & set up the new dashboard in the DSS.
Pre-requisites
Before you proceed with the configuration, make sure the following pre-requisites are met -
Prior knowledge of Spring boot
Prior knowledge of Kafka
Prior knowledge of Elastic Search
Prior knowledge of Kibana
Prior knowledge of EQL (Elastic Query Language)
Prior knowledge of JSON
Key Functionalities
Creating a DSS dashboard schema
DSS ingest service APIs
Ingest service configurations
Creating a Kafka sync connector to push the data to elastic search
1. Creating a DSS dashboard schema
When we are going to start indexing the DSS collection v2 index. We should create the schema in the ES using the Kibana query as there in the below file.
2. DSS ingest service API
3. Ingest service configurations
Transform collection schema for V2
This transform collection v1 configuration file is used to map the incoming data. This mapped data will go inside the data object in the DSS collection v2 index.
Here: $i, the variable value that gets incremented for the number of records of paymentDetails.
$j, the variable value that gets incremented for the number of records of billDetails.
Enrichment Domain Configuration
This configuration defines and directs the Enrichment Process that the data goes through.
For example, if the data which is incoming is belonging to a Collection Module data, then the Collection Domain config is picked. And based on the Business Type specified in the data, the right config is picked.
In order to enhance the data of Collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set.
Topic Context Configuration
Topic Context Configuration is an outline to define which data is received on which Kafka Topic.
Indexer Service and many other services are sending out data on different Kafka Topics. If the Ingest Service is asked to receive those data and pass it through the pipeline, the context and the version of the data being received have to be set. This configuration is used to identify as in which Kafka topic consumed the data and what is the mapping for that.
JOLT Domain Transformation Schema
JOLT is a JSON to JSON Transformation Library. In order to change the structure of the data and transform it in a generic way, JOLT has been used.
While the transformation schemas are written for each Data Context, the data is transformed against the schema to obtain transformed data.
Validator Schema
Validator Schema is a configuration Schema Library from Everit Bypassing the data against this schema, it ensures that the data abides by the rules and requirements of the schema which has been defined.
Enhance Domain configuration
This configuration defines and directs the Enrichment Process that the data goes through.
For example, if the Data which is incoming is belonging to a Collection Module data, then the Collection Domain Config is picked. And based on the Business Type specified in the data, the right config is picked and the final data is placed inside the domain object.
In order to enhance the data of Collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set.
For the Kafka connect to work, Ingest pipeline application properties or in environments direct push must be disabled.
es.push.direct=false
If DSS collection index data is indexing directly ( without Kafka connector) to ES through the ingest pipeline then, make the application properties or in environments, direct push must be enabled.
es.push.direct=true
4. Creating a Kafka sync connector to push the data to the Elasticsearch
Configure the Kafka topics in the environments or Ingest pipeline application properties as shown below.
To start the indexing we will create a connecter that will take data from the topic and push it to the index we have mentioned in the "transforms.TopicNameRouter.replacement" and mention the ES host in the Kafka connection we have to mention the host URL in “connection.url”.
To create the Kafka connector run the below curl command inside the playground pod:
