TL Renewal Alert & Owner Type Test Cases

PT UI Audit Test Cases

PGR Enhancement Test Cases

PT Common Search & TL Integration Test Cases

Notifications Based On Channel Test Cases

Update Mobile Number Feature Test Cases

National Dashboard & DSS Test Cases

Birth & Death Test Cases

Download Our Test Cases

Mission

Building an urban governance platform to catalyze digital transformation across towns and cities - changing the way citizens interact with government bodies.

Overview

DIGIT Urban Stack is a set of Open APIs, services, and reference implementations, set up as a public good, to allow government entities, businesses, startups, and civil society to use a unique digital Infrastructure and build solutions for urban India at a large scale. It provides a set of open standards, specifications and documentation to create a level playing field and enable ecosystem players to innovate on the stack. As a public good, the platform is provided without profit or restriction to all members of society.

Goals

The urban mission offers digital governance solutions that encapsulate the core platform principles. The apps provide

• Secure & reliable governance tools

• Simple reusable modules for effective integration

• Scalable & standardized solutions

• Open APIs to promote interoperability

• Multiple channel support

• Configurable building blocks that support customization at each stage

Approach

DIGIT-Urban focuses on inclusion and is designed on the principle of enhancing both platform openness and choice for citizens. The platform uses open APIs and standards, creating a powerful framework to drive convergence across the multiple systems currently in use and to lower the barrier to entry for locally-developed solutions.

Keeping in mind that most Indians use the internet through their phones, we follow and advocate a “mobile-first” approach, while supporting multi-channel access to accommodate diverse needs and preferences.

• Open Source - DIGIT is Open Source and has been built using the best in class Open Source technology stacks powering the most advanced companies in the world. DIGIT is able to provide the lowest total cost of ownership and helps ensure that governments retain strategic control of their systems and data.

• Mobile-Enabled - DIGIT has adopted a mobile-first approach, with robust mobile applications enabling citizens to easily access government services through their phones. Government officials and field workers are also empowered with mobile applications that enable them to deliver 24×7 governance with ease.

• Real-time Dashboards - DIGIT’s real-time dashboards provide governments with actionable insights on demand. Administrators and department heads are empowered with verified data that enables them to manage their work and budgets better. Field-level employee reports enable effective performance management and ensure efficient usage of government resources.

Useful Links

Contact Us

MDMS Changes

Config Changes

Infra Changes

QA Test Cases

UAT Regression Test Cases

Overview

The Birth and Death module provides a digital interface, allowing citizens to search for and download the Birth and Death certificate. Also, the correctness of the certificate can be verified by scanning the QR code. The module enables the following:

• Reduced physical touchpoints as citizens are not required to visit offices

• Reduced time with the instantaneous download of certificates

• Verification of the correctness of the certificate by scanning the QR Code

The Birth & Death module also enables ULB employees to create new registrations and also search for any previous applications through the employee side interface of the module.

In addition to the module, the DSS(Decision Support System) at the state level and national level instances enables administrators to view the records and have an aggregate view of the certificate downloads and other metrics for research and analysis purposes. The demographic data of the birth and death records also helps administrators to detect anomalies and make data-driven decisions.

Release Highlights

In this release of the Birth & Death module, the following modules were developed

1. Birth & Death Module

   • Employee side interface

   • Citizen side interface

   • User access for

     1. Birth record creator - Can only create new applications

     2. Death record creator - Can only create new applications

     3. Birth record viewer - Can only search and view applications

     4. Death record viewer - Can only search and view applications

     5. Birth record editor - Can edit existing applications

     6. Death record editor - Can edit existing applications

     7. DSS viewer - View DSS

2. Birth & Death State DSS

3. Birth & Death National DSS

Release Features

Birth & Death Module

For an employee, the B&D module gives access to create birth records and death applications based on the registration details that they receive from the hospital, morgue, or from concerned authorities. They can also search the existing records of birth and death registrations based on date ranges, view them in tabular forms, and take printouts.

For a citizen, the Birth & Death module offers the convenience of downloading birth and death certificates using a simple search interface. They can also make payments if required for downloading the certificates, view past records and payments, and also download payment receipts.

Birth & Death Module - State DSS

The Birth & Death module shows the aggregated values for the certificate downloads and payment collections on the overview page of state DSS. Furthermore, in the individual module page, it shows

1. Distinct birth and death tabs that display metrics and charts for each segment

2. Number of certificate downloads and collection from payments

3. Certificate download trend across time

4. Chart showing certificate downloads by channel

5. Number of delayed registrations where the date of registration and date of birth/death differs by more than a year

6. Chart showing births and deaths by gender

7. Chart showing deaths by age category

The module also has the capability to show the ULB and ward-wise drill down for the delayed registrations and downloads by channel

Birth & Death Module - National DSS

The Birth & Death module shows the aggregated values of the certificate downloads and payment collections on the overview page of the National DSS. The application numbers and collection amounts are also aggregated and shown in the Total Collection and Total applications metrics on the Landing Page. Furthermore, in the individual module page, it shows

1. Distinct birth and death tabs that display metrics and charts for each segment

2. Number of certificate downloads and collection from payments

3. Certificate download trend across time

4. Chart showing certificate downloads by channel

5. Number of delayed registrations where the date of registration and date of birth/death differs by more than a year

6. Chart showing births and deaths by gender

7. Chart showing deaths by age category

The module also has the capability to show the ULB and ward-wise drill down for the delayed registrations and downloads by channel

Known Issues

None.

Upcoming Release Features

1. Birth & Death Report

2. User roles for - Birth Report viewer, Death Report Viewer

Reference Doc Links

Overview

This release provides features which are identified as GAP between old UI and New UI.

Release Highlights

UI audit of property tax covers the features given below.

Citizen

1. Property Registration

2. Property Update

3. Search Property

4. Search Application

5. My Payments

6. Search and Pay

7. View Property

8. View Mutation Application

9. Pay Mutation Fee

Employee

1. View Payment History

2. Search Property

3. Search Application

4. Assess Property

5. View Mutation Application

6. Collect Mutation Fee

7. Update Property

Release Features

Known Issues

None.

Upcoming Release Features

1. Reports revamp

2. Output Documents PDF revamp

Reference Doc Links

Release Summary

This note consists of common UI/UX elements revamped for employee and citizen Profiles.

New ‌Feature Additions

Detailed user Manuals

Common UI/UX:

Release Summary

DIGIT 2.7 release has got new modules, a few functional changes, and non-functional changes.

Functional changes

• National Dashboard for PT, TL, OBPS, PGR, W&S, mCollect, and Fire NOC;

• State level DSS of OBPS, W&S, Fire NOC and mCollect;

• PT UI/UX Audit Feedbacks, TL UI/UX Audit Feedbacks, PGR UI/UX Enhancements Audit Feedbacks;

• PT common Search, PT lightweight create and integrate with TL;

• Notification for different channels - W&S, PT, and TL;

• Common UI/UX - Citizen Profile, Employee Profile;

• Multi Tenancy Selection, Birth and Death module, State DSS and National Dashboard for Birth and Death

Non-functional changes

• National Dashboard Ingest API and NUGP STQC Security fixes

New ‌Feature Additions

Enhancements

‌Document Resources and Links

UI Technical Documents

Backend Service Documents

Overview

In this release of the National Urban Dashboard, both state and national level DSS instances were developed for some modules and only the national level instance for a few others.

Release Highlights

All the module dashboards have the following common features:

• Filters for the date range, States, ULBs

• Denominations to change the revenue metrics view

• Drill down feature in the service report where state wise, ULB wise, and ward wise reports are available by clicking on the boundary name

• Definitions for all the metrics and chart elements are viewable by hovering over the desired element/metric

• The search feature in the filter boxes and service tabular reports

• Download, share (WhatsApp, email) for individual cards and overall dashboard

Each module dashboard presents the aggregated urban data from the ward, ULB, and state levels in concise charts, graphs, and also in raw numbers as per the KPIs defined. Except for Landing Page, all the other modules are similar for both national and state-level instances.

Click on the file link below to find the detailed definitions of each KPI in the dashboard.

Release Features

The individual dashboard features are listed in the table below:

Known Issues

None.

Upcoming Release Features

• Sidebar navigation

• About, Purpose pages

Reference Doc Links

Release Summary

FSM 1.1 is a release that has a few functional changes.

• Functional: Pre-pay and post-pay service, multi-trip, capturing gender Information, desludging request flow enhancements, FSTPO vehicle log flow enhancements, DSO flow enhancements, and FSM generic enhancements.

New ‌Feature Additions

Enhancements

Document Resources and Links

UI Technical Documents

Backend Service Documents

DIGIT is an open-source platform licensed under the MIT license () compliant with the NUIS digital blueprint.

Detailed mapping of DIGIT’s capabilities with the core requirements mentioned in the NUIS digital blueprint has been done below:

Interoperability

Data specifications/models are available for domain entities. DIGIT is designed as an API-first platform wherein data specs/models are created for all key entities thus ensuring interoperability through open APIs and open standards. Taxonomies are available for the key domain entities/registries. These can later be harmonised with standard taxonomies in the domain as and when they are made available.

DIGIT data models and APIs are published as Open APIs freely available to everyone in the ecosystem. In Punjab, the DIGIT module was easily integrated with 3rd party payment apps like Paytm, Airtel Money, BBPS, etc to increase citizen access and improve collections. At present, DIGIT provides at least 3 key distinct APIs for all domain entities - create, update and search.

Deactivation/Cancellation of key entities in DIGIT is achieved through updating their status to inactive as per their defined specification/API contracts. Given the API-first and micro-services-driven nature of DIGIT, current APIs and models can be quickly harmonised with national standards as and when they are made available. DIGIT strives to leverage established domain standards (national/international) wherever available.

Data Privacy And Security By Design

Data Privacy

Data privacy capabilities are available to mark and protect sensitive data. The core service layer of DIGIT includes signing and encryption service as one of the core services that provide capabilities to sign/encrypt/mask sensitive data. It is designed such that it can work against software key stores and can be extended to integrate with any kind of hardware key store to store and protect signing and encryption keys.

Encryption requirements can be defined and adhered to for the storage of sensitive data. DIGIT requires the User PII data to be stored in its User service which is by default enabled for encryption of sensitive data as User Data Vault. All other services in DIGIT are required to access PII data by explicitly calling the User service - which in turn audits all access to PII. In addition, individual services in DIGIT can leverage DIGIT’s signing and encryption service (which is what User Service leverages to create User Data Vault) to further protect additional sensitive data available with the services.

DIGIT provides the capability to define workflows for data modification that can be configured to have approval steps to get needed consent for any data modification activities. DIGIT currently provides RBAC (Role-Based Access Control) based access control for access (search) to data.

Security

Appropriate access controls can be defined in the APIs to ensure authorised access to sensitive data. DIGIT is designed to handle authentication and authorisation as perimeter control at its API gateway layer to ensure unauthorised calls are not allowed to even contact the respective micro-services. DIGIT provides an RBAC (Role-Based Access Control) mechanism where users are explicitly provided access to relevant resources by assigning them appropriate roles. By default, DIGIT supports OAUTH-based authentication for individual users and APIs. However, the Authentication and Authorization filter on DIGIT is designed to be easily extendable to support any further Auth and Auth needs.

Perimeter security mechanism in DIGIT also helps developers in focusing on the functional developments in further services and offloading the access control requirements for new resources and their APIs to the API gateway using simple configurations.

DIGIT also ensures that risks like the following are taken care of:

• Privilege escalation – form field manipulation

• Failure to restrict URL access

• Insecure direct object references (IDOR)

• Malicious file upload leads to cross-site scripting

• Improper authentication

• Missing account lockout

• Request throttling attack

• Weak encoding mechanism

• Sensitive information in URL

• Lack of automatic session expiration

• Insecure banner implementation

• Concurrent session

• Clickjacking

• Improper error handling

Transparency And Accountability Through Data

DIGIT has the capability to define key registries in OpenAPI 3.0 specs formats and easily achieve key APIs like create/update/search using its building blocks in core services mainly through configurations and using lightweight extensions on a needs basis.

DIGIT has the capability to protect person-specific sensitive data by encrypting them in the user data vault (User Registry) which allows configuration-based protection of sensitive PII. DIGIT requires additional registries to reference PII using this mechanism. In addition, registries in DIGIT can leverage its data protection (Signing and Encryption) core service to provide additional protection to registry-specific attributes.

Registry data in DIGIT can be signed for tamper-proofing using its signing and encryption core service. A proof of concept for this has already been done on the ePass module that was built on the DIGT platform. All key data modifications in DIGIT are access logged to provide an audit trail, which can be accessed through APIs. The upcoming version of DIGIT is planning to bring in the concept of immutable event logs to further strengthen this capability. DIGIT leverages open-source telemetry to provide the ability to gather telemetry data and extends it for the DIGIT-specific processing pipeline. This framework allows for additional event definitions and contextual extension of the telemetry processing pipeline thereby future-proofing this capability in DIGIT.

Reusability and Extensibility

DIGIT platform is designed as a collection of more than 50+ atomic microservices which are bundled together in a given context to provide end solutions. Microservices in DIGIT can be mainly categorized in three categories: Data services (Registries, reference Master data management, etc.), Tech infrastructure services (Authentication, authorisation, notification engine etc.) and domain services (Assessment, NOC etc.). Citizen, employee and administrative interfaces in DIGIT use these microservices to achieve the needed functionality.

Data models and APIs in DIGIT are defined as OpenAPI 3.0 specifications and can be extended by using a combination of configuration and extension techniques. E.g. if the additional attributes are only needed to be stored with format validation, it can be a simple schema extension, while if the additional business checks/functionality need to be implemented using the extended attributes then it can be achieved using pre/post request filters or extending underlying microservices.

DIGIT allows extension of existing capabilities without needing architectural interventions. As described above extension of existing functionality on DIGIT can be achieved using additional configurations, additional extension services or request/response filters.

Several partners have extended DIGIT modules to cater to new use cases. For instance DIGIT mCollect module caters to collection of fees for more than 50 services on the counter, but it did not have a citizen interface for payment of these services online. Directorate General Defence Estates (DGDE) wanted to introduce this interface for the citizen’s of cantonment boards in India and were able to easily enhance the mCollect module to include this capability. Similarly Punjab has reused several DIGIT core services to develop new modules on the platform with minimum efforts.

DIGIT supports single-instance multi-tenancy to enable sharing of the underlying infrastructure, also all DIGIT data models and services are designed to be multi-tenanted.

DIGIT uses API first approach in its design and development to ensure loose coupling between its various components. These APIs are clearly defined using OpenAPI 3.0 specifications to ensure clear documentation.

Evolvability and Scale

As described above, extension of existing functionality on DIGIT can be achieved using additional configurations, additional extension services or request/response filters. Similarly new functionality can be added by rebundling existing building blocks in context of new use cases and implementing only additionally required services without requiring any architectural overhaul. Additionally due to its loosely coupled API driven design DIGIT allows for new components to be implemented in the technology that is most useful for that use case.

API driven, microservices based architecture of DIGIT enables its components to evolve separately. On DIGIT Individual components can evolve separately to enable heterogeneous evolution of the system.

DIGIT uses SemVer 2.0 for versioning of its microservices and interfaces. Semantic versioning is a formal convention for specifying compatibility using a three-part version number: major version; minor version; and patch. More details on this can be found on this link: https://semver.org/.

DIGIT is designed to be horizontally scalable. Microservices based architecture of DIGIT also enables it to scale only needed components/services, thereby providing resource efficiency. E.g. Billing and Collection services can be scaled separately during financial year closing if the load pattern indicates increasing volume of bill payments during that period.

DIGIT is designed to be hardware agnostic and can be run on any hardware. It has been tested on multiple commercial clouds and state sponsored bare metal infrastructure. Components of DIGIT that need to use underlying hardware have been carefully chosen (in case where DIGIT is using other open source components) or designed (DIGIT’s own components) to provide a layer of abstraction that can be extended for any types of hardware.

Multi-channel Access

DIGIT is designed using API first approach, therefore enabling any user interface channel to leverage it. DIGIT’s own user interfaces (Web/mobile app, WhatsApp chatbot) are implemented using its APIs to ensure offered platform capabilities and data are accessible to any delivery channel based on configured policies. In states like Punjab and AP where DIGIT modules are being used, the citizens have been given multi-channel access - ULB counters, Web portals, Mobile App, WhatsApp Chatbot and 3rd party applications like Paytm, BBPS to avail local government services.

DIGIT’s access control mechanism can be configured to provide different levels of access based on channels and roles.

Ecosystem-driven

DIGIT platform and its user interfaces are completely open source. Also, all external components used in DIGIT are also Open Source. Due to its API based and event driven architecture DIGIT can be integrated with any existing stack. Wherever appropriate, DIGIT also provides out of box integrations with crucial stacks/platforms. The most common integrations are to payment gateways, SMS providers and SMTP email servers for a typical implementation.

More than 14 organizations have already partnered with us to implement DIGIT across multiple implementations in the country and have built more than 20 new solutions on top of the platform.

DIGIT also provides the capability to gather feedback from the ecosystem in a digital manner. Feedback capability in DIGIT can be looked at the following levels:

• Service Delivery feedback on services offered through DIGIT - DIGIT provides a highly configurable and extensible Public Grievance module to enable this kind of feedback/redressal for functional users (Citizens, employees etc)

• Service Usage feedback - DIGIT user interfaces include a telemetry SDK which is backed by telemetry infrastructure in DIGIT platform. Coupled with API access logs, this enables DIGIT to gather usage feedback through live action and can be used for fine tuning interfaces and APIs

• Design/Feature feedback - As an open source project on github, DIGIT provides a mechanism to provide comments/feedback on its various components using github. This feedback can be leveraged to create a Point of View on the future roadmap for the platform.

DIGIT is India’s largest open-source platform for digital governance. It is built on OpenAPI (OAS 2.0) and provides API-based access to a variety of urban/municipal services enabling state governments and city administrators to provide citizen services with relevant new services and also integrating the existing system into the platform and run seamlessly on any commercial/on-prem cloud infrastructure with scale and speed.

Key Architecture Highlights

• DIGIT is a microservices-based platform that is built to scale. Microservices are small, autonomous and developer-friendly services that work together.

• A big software or system can be broken down into multiple small components or services. These components can be designed, developed & deployed independently without compromising the integrity of the application.

• Parallelism in development: Microservices architectures are mainly business-centric.

• MicroServices have smart endpoints that process info and apply logic. They receive requests, process them, and generate a response accordingly.

• Decentralized control between teams, so that its developers strive to produce useful tools that can then be used by others to solve the same problems.

• MicroServices architecture allows its neighbouring services to function while it bows out of service. This architecture also scales to cater to its clients’ sudden spike in demand.

• MicroService is ideal for evolutionary systems where it is difficult to anticipate the types of devices that may be accessing our application.

Multi-layer Architecture

DIGIT follows Multilayer or n-tiered distributed architecture pattern. As seen in the illustration above there are different horizontal layers with some set of components eg. Data Access Layer, Infra Services, Business Services, different modules layers, client Apps and some vertical adapters. Every layer consists of a set of microservices. Each layer of the layered architecture pattern has a specific role and responsibility within the application.

• Layered architecture increases flexibility, maintainability, and scalability

• Multiple applications can reuse the components

• Parallelism

• Different components of the application can be independently deployed, maintained, and updated, on different time schedules

• Layered architecture also makes it possible to configure different levels of security to different components

• Layered architecture also helps users test the components independent of each other

about , , , , , and .

This section contains docs and information resources that guide you through the key DevOps concepts and its role in managing the DIGIT platform.

All content on this page by is licensed under a .

Details coming soon...

Overview

DIGIT is the largest urban governance platform built for billions and billions of transactions between citizens and the state govt through various municipal services/integration. The platform is built with key capabilities like scale, speed, integration, configurable, customizable, extendable, multi-tenanted, security, etc. Here, we shall discuss the key requirements and capabilities.

Pre-requisites

Before proceeding to set up DIGIT, it is essential to know some of the key technical details about DIGIT, like architecture, tech stack and how it is packaged and deployed on various infrastructures. Some of these details are explained in the previous sections. Below are some of the key capabilities to know about DIGIT as a platform.

1. DIGIT is a collection of various services built as RESTFul APIs with OpenAPI standard

2. DIGIT is built as MSA (Microservices Architecture)

3. DIGIT services are packaged as containers and deployed as docker Images.

4. DIGIT is deployed on Kubernetes which abstracts any Cloud/Infra suitable and standardised for DIGIT deployment.

5. DIGIT deployment, configuration and customization are done through Helm Charts.

6. Kubernetes cluster setup is done through code like terraform/ansible suitably.

Why OpenAPI

The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for REST APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenAPI, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. Similar to what interface descriptions have done for lower-level programming, the OpenAPI Specification removes the guesswork in calling a service.

Why Microservice Architecture

Microservices are nothing but breaking big beasts into smaller units that can independently be developed, enhanced and scaled as a categorized and layered stack that gives better control over each component of an application that exists in its own container, independently managed and updated. This means that developers can build applications from multiple components and program each component in the language best suited to its function, rather than having to choose a single less-than-ideal language to use for everything. Optimizing software all the way down to the components of the application helps you increase the quality of your products. No time and resources are wasted managing the effects of updating one application on another.

Why Containerized/Dockerized

Comparatively the best infra choice for running a microservices application architecture is application containers. Containers encapsulate a lightweight runtime environment for the application, presenting a consistent environment that can follow the application from the developer's desktop to testing to final production deployment, and you can run containers on cloud infra with physical or virtual machines.

Why Kubernetes

As most modern software developers can attest, containers have provided us with dramatically more flexibility for running cloud-native applications on physical and virtual infrastructure. Kubernetes allows you to deploy cloud-native applications anywhere and manage them exactly as you like everywhere. For more details refer to the above link that explains various advantages of Kubernetes.

Why Helm Charts

Kubernetes, the popular container orchestration system, is used extensively. However, it can become complex: you have to handle all of the objects (ConfigMaps, pods, etc.), and would also have to manage the releases. Both can be accomplished with Kubernetes Helm. It is a Kubernetes package manager designed to easily package, configure, and deploy applications and services onto Kubernetes clusters in a standard way, this helps the ecosystem to adopt the standard way of deployment and customization.

For being successful in the DIGIT Setup, below are certain requirements that need to be ascertained:

Skills Needed

• On-premise/private cloud accounts

  • Interface to access and provision required infra

  • In the case of SDC, NIC or private DC, it'll be VPN to an allocated VLAN

  • SSH access to the VMs/machines

• Infra Skills

  • Public cloud

    • Managed Kubernetes services like AKS or EKS or GKE on Azure, AWS and GCP respectively

  • Private Clouds (SDC, NIC)

    • Clouds like VMware, OpenStack, Nutanix and more, may or may not have Kubernetes as a managed service. If yes we may have to estimate only the worker nodes depending on the number of ULBs and DIGIT's municipal services that you opt.

    • In the absence of the above, you have to provision the Kubernetes cluster from the plain VMs as per the general Kubernetes setup instruction and add worker nodes.

• Operations Skills

  • Understanding of Linux, containers, VM Instances, Load Balancers, Security Groups/Firewalls, Nginx, DB Instance, Data Volumes

  • Experience with Kubernetes, Docker, Jenkins, Helm, Infra-as-code, Terraform

  • Experience in DevOps/SRE practice on microservices and modern infrastructure

High-level Action To Deploy DIGIT

1. Provisioning the Kubernetes Cluster in any of the

   • Commercial cloud or

   • Private State datacenter (SDC) or

   • National Cloud (NIC)

2. Setting up the persistent disk volumes to attach to DIGIT backbone stateful containers like

   • ZooKeeper

   • Kafka

   • Elastic Search

3. Setting up the Postgres DB

   • On a public cloud, provision a Postgres RDS instance.

   • Private cloud, provision a Postgres DB on a VM with the backup, HA/DRS

4. Preparing deployment configuration for required DIGIT services using Helm templates from the InfraOps like the following

   • Preparing DIGIT service helm templates to deploy on the Kubernetes cluster

   • K8s Secrets

   • K8s ConfigMaps

   • Environment variables of each microservices

5. Deploy the stable released version of DIGIT and the required services

6. Setting up Jenkins job to build, bake images and deploy the components for the rolling updates

7. Setup Application monitoring, Distributed Tracing, Alert management

Hiring DevOps Resources

Anyone involved in hiring DevOps engineers will realize that it is hard to find prospective candidates who have all the skills listed in this section.

Ultimately, the skill set needed for an incoming DevOps engineer would depend on the current and short-term focus of the operations team. A brand new team that is rolling out a new software service would require someone with good experience in infrastructure provisioning, deployment automation, and monitoring. A team that supports a stable product might require the service of an expert who could migrate home-grown automation projects to tools and processes around standard configuration management and continuous integration tools.

DevOps practice is a glue between engineering disciplines. An experienced DevOps engineer would end up working in a very broad swath of technology landscapes that overlaps with software development, system integration, and operations engineering.

An experienced DevOps engineer would be able to describe most of the technologies that is described in the following sections. This is a comprehensive list of DevOps skills for comparing one’s expertise and a reference template for acquiring new skills.

In theory, a template like this should be used only for assessing the current experience of a prospective hire. The needed skills can be picked up on the jobs that demand deep knowledge in certain areas. Therefore, the focus should be to hire smart engineers who have a track record of picking up new skills, rolling out innovative projects at work, and contributing to reputed open-source projects.

1. Knowledge of Infrastructure

A DevOps engineer should have a good understanding of both classic (data centre-based) and cloud infrastructure components, even if the team has a dedicated infrastructure team.

A. Classic Infrastructure

This involves how real hardware (servers and storage devices) are racked, networked, and accessed from both the corporate network and the internet. It also involves the provisioning of shared storage to be used across multiple servers and the methods available for that, as well as infrastructure and methods for load balancing.

Virtualization Basics

• Hypervisors.

• Virtual machines.

• Object storage.

• Running virtual machines on PC and Mac (Vagrant, VMWare, etc.).

B. Cloud Infrastructure

Cloud infrastructure has to do with core cloud computing and storage components as they are implemented in one of the popular virtualization technologies (VMWare or OpenStack). It also involves the idea of elastic infrastructure and options available to implement it.

Networking Basics

• Network layers

• Routers, domain controllers, etc.

• Networks and subnets

• IP address

• VPN

• DNS

• Firewall

• IP tables

• Network access between applications (ACL)

• Networking in the cloud (i.e., Amazon AWS)

Load Balancing

• Load balancing infrastructure and methods

• Geographical load balancing

• Understanding of CDN

• Load balancing in the cloud

2. DevOps Toolchain

A DevOps engineer should have experience using specialized tools for implementing various DevOps processes. While Jenkins, Dockers, Kubernetes, Terraform, Ansible, and the like are known to most DevOps guys, other tools might be obscure or not very obvious (such as the importance of knowing one major monitoring tool in and out). Some tools like, source code control systems, are shared with development teams.

The list here has only examples of basic tools. An experienced DevOps engineer would have used some application or tool from all or most of these categories.

Source Code Management (SCM) System

• Expert-level knowledge of an SCM system such as Git or Subversion.

• Knowledge of code branching best practices, such as Git-Flow.

• Knowledge of the importance of checking in Ops code to the SCM system.

• Experience using GitHub.

Bug Management System

• Experience using a major bug management system such as Bugzilla or Jira.

• Ability to have a workflow related to the bug filing and resolution process.

• Experience integrating SCM systems with the bug resolution process and using triggers or REST APIs.

Collaborative Documentation System

• Knowledge of Wiki basics.

• Experience using MediaWiki, Confluence, etc.

• Knowledge of why DevOps projects have to be documented.

• Knowledge of how documents were organized on a Wiki-based system.

Build and CI

• Experience building on Jenkins standalone, or dockerized.

• Experience using Jenkins as a Continuous Integration (CI) platform.

• CI/CD pipeline scripting using groovy

• Experience with CI platform features such as:

  • Integration with SCM systems.

  • Secret management and SSH-based access management.

  • Scheduling and chaining of build jobs.

  • Source-code change based triggers.

  • Worker and slave nodes.

  • REST API support and Notification management.

Artefacts Management

• Should know what artefacts are and why they have to be managed.

• Experience using a standard artefacts management system such as Artifactory.

• Experience caching third-party tools and dependencies in-house.

Configuration Management

• Should be able to explain configuration management.

• Experience using any Configuration Management Database (CMDB) system.

• Experience using open-source tools such as Cobbler for inventory management.

• Ability to do both agent-less and agent-driven enforcement of configuration.

• Experience using Ansible, Puppet, Chef, Cobbler, etc.

Orchestration and Deployment

• Knowledge of the workflow of released code getting into production.

• Ability to push code to production with the use of SSH-based tools such as Ansible.

• Ability to perform on-demand or Continuous Delivery (CD) of code from Jenkins.

• Ability to perform agent-driven code pull to update the production environment.

• Knowledge of deployment strategies, with or without an impact on the software service.

• Knowledge of code deployment in the cloud (using auto-scaling groups, machine images, etc.).

Monitoring

• Knowledge of all monitoring categories: system, platform, application, business, last-mile, log management, and meta-monitoring.

• Status-based monitoring with Nagios.

• Data-driven monitoring with Zabbix.

• Experience with last-mile monitoring, as done by Pingdom or Catchpoint.

• Experience doing log management with ELK.

• Experience monitoring SaaS solutions (i.e., Datadog and Loggly).

3. System Tools and Methods

To get an automation project up and running, a DevOps engineer builds new things such as configuration objects in an application and code snippets of full-blown programs. However, a major part of the work is glueing many things together at the system level on the given infrastructure. Such efforts are not different from traditional system integration work and, in my opinion, the ingenuity of an engineer at this level determines his or her real value on the team. It is easy to find cookbooks, recipes, and best practices for vendor-supported tools, but it would take experience working on diverse projects to gain the necessary skill set to implement robust integrations that have to work reliably in production.

Important system-level tools and techniques are listed here. The engineer should have knowledge about the following.

Access Management

• Users and groups on Linux.

• Use of service accounts for automation.

• Sudo commands, /etc/sudoers files, and passwordless access.

• Using LDAP and AD for access management.

• Remote access using SSH.

  • SSH keys and related topics.

  • SCP, SFTP, and related tools.

  • SSH key formats.

• Managing access using configuration management tools.

Password Management

• Use of GPG for password encryption.

• Tools for password management such as KeePass.

• MD5, KMS for encryption/decryption.

• Remote access with authentication from automation scripts.

• Managing API keys.

• Jenkins plugins for password management.

Build

• Basics of compilers such as node.js and Javac.

• Make and Makefile, npm, Maven, Gradle, etc.

• Code libraries in Node, Java, Python, React etc.

• Build artefacts such as JAR, WAR and node modules.

• Running builds from Jenkins.

Packaging

• Packaging files: ZIP, TAR, GZIP, etc.

• Packaging for deployment: RPM, Debian, DNF, Zypper, etc.

• Packaging for the cloud: AWS AMI, VMWare template, etc.

• Use of Packer.

• Docker and containers for microservices.

4. Artefacts Management

• Use of artefacts repository: Distribution and release of builds; meeting build and deployment dependencies

• Serving artefacts from a shared storage volume

• Mounting locations from cloud storage services such as AWS S3

• Artifactory as artefacts server

File Transfer

• SCP, Rsync, FTP, and SSL counterparts

• Via shared storage

• File transfer with cloud storage services such as AWS S3

Deployment

• Code pushing using system-level file transfer tools.

• Scripting using SSH libraries such as Paramiko.

• Orchestrating code pushes using configuration management tools.

Job Management

• Use of crontab.

• Running jobs in the background; use of Nohup.

• Use of screen to launch long-running jobs.

• Jenkins as a process manager.

Files and Storage

• Typical uses of the find, DF, DU, etc.

Linux Distributions

• A comparison of popular distributions.

• Checking OS release and system info.

• Package management differences.

• OS Internals and Commands

Text Processing

• Typical uses of SED, AWK, GREP, TR, etc.

• Scripting using Perl, Python.

• Regular expressions.

• Support for regular expressions in Perl and Python.

Troubleshooting Toolkit

Sample usages and steps to install these tools:

• NC

• Netstat

• Traceroute

• VMStat

• LSOF

• Top

• NSLookup

• Ping

• TCPDump

• Dig

• Sar

• Uptime

• IFConfig

• Route

Programming Primer for DevOps

One of the attributes that helps differentiate a DevOps engineer from other members in the operations team, like sysadmins, DBAs, and operations support staff, is his or her ability to write code. The coding and scripting skill is just one of the tools in the DevOps toolbox, but it's a powerful one that a DevOps engineer would maintain as part of practising his or her trade.

Coding is the last resort when things cannot be integrated by configuring and tweaking the applications and tools that are used in an automation project.

Scripting

Bash Scripting Essentials

Many times, a few lines of bash script could be the best glue code integrating two components in the whole software system. DevOps engineer should have basic shell scripting skills and Bash is the most popular right now.

Python

If a script has to deal with external systems and components or it's more than just a few lines of command-lines and dealing with fairly complex logic, it might be better to write that script in an advanced scripting language like Python, Perl, or Ruby.

Knowledge of Python would make your life easier when dealing with DevOps applications such as Ansible, which uses Python syntax to define data structures and implement conditionals for defining configurations.

Web Programming

One of the categories of projects a DevOps engineer would end up doing is building dashboards. Though dashboarding features are found with most of the DevOps tools, those are specific to the application, and there will be a time when you may require to have a general-purpose dashboard with more dynamic content than just static links and text.

Another requirement is to build web UI for provisioning tools to present those as self-service tools to user groups.

In both these cases, deep web programming skills are not required. Knowledge of a web programming friendly language such as PHP and a JavaScript/CSS/HTML library like Composer would be enough to get things started. It is also important for the DevOps engineer to know the full-stack, in this case, LAMP, for building and running the web apps.

Configuration Languages

Almost every application and tool that is used for building, deploying, and maintaining software systems use configuration files. While manual reading of these files might not require any expertise, a DevOps engineer should know how config files in such formats are created and parsed programmatically.

A DevOps engineer should have a good understanding of these formats:

• INI.

• XML.

• JSON.

• YAML.

The engineer should also know how these formats are parsed in his/her favourite scripting language.

REST API

The wide acceptance of REST API as a standard to expose features that other applications can use for system integration made it a feature requirement for any application that wants to be taken seriously. The knowledge of using REST API has become an important skill for DevOps engineer.

• HTTP/HTTPS: REST APIs are based on HTTP/HTTPS protocol and a solid understanding of its working is required. Knowledge of HTTP headers, status codes, and main verbs GET, POST, and PUT.

• REST API basics: Normal layout of APIs defined for an application.

• Curl and Wget: Command-line tools to access REST API and HTTP URLs. Some knowledge of the support available for HTTP protocol in scripting languages will be useful and that would be an indication of working with REST APIs.

• Authentication methods: Cookie-based and OAuth authentication; API keys; use of If-Match and If-None-Match set of HTTP headers for updates.

• API management tools: If the application you support provides an API for the users, most probably, its usage will be managed by some API Gateway tool. Though not an essential skill, experience in this area would be good if one works on the API provider side.

Programming With Data Repositories

There was a time when mere knowledge of programming with RDBMS was enough for an application developer and system integrator to manage application data. Now with the wide adoption of Big Data platform like Hadoop and NOSQL systems to process and store data, a DevOps engineer needs varied requirements, from one project to another. Core skills are the following:

• RDBMS: MySQL, Postgres, etc. knowledge of one or more is important.

• Setting up and configuring PostGres: As an open-source database used with many other tools in the DevOps toolchain, consider this as a basic requirement for a DevOps engineer. If one hasn’t done this, he or she might not have done enough yet.

• Running queries from a Bash script: How to run a database query via a database client from a Bash script and use the output. MySQL is a good example.

• Database access from Perl/PHP/Python: All the major scripting languages provide modules to access databases and that can be used to write robust automation scripts. Examples are Perl DBI and Python’s MySQLdb module.

• DB Backups: Migration, Logging, monitoring and cleanup.

Programming for Cloud

Those who have built cloud infrastructure with a focus on automation and versioning should know some of these (or similar) tools:

• cloud-init: Cloud-init can be used to configure a virtual machine when it is spun up. This is very useful when a node is spun up from a machine image with baseline or even application software already baked in.

• AWS/Azure/GCloud CLI: If the application runs on Commercial cloud, knowledge of CLI is needed, which would be handy to put together simple automation scripts.

• Terraform: HashiCorp’s Terraform is an important tool if the focus would be to provision infrastructure as code (IaaS). Using this, infrastructure can be configured independently of the target cloud or virtualization platform.

• Ansible: It can be used to build machine images for a variety of virtualization technologies and cloud platforms, it is useful if the infrastructure is provisioned in a mixed or hybrid cloud environment.

Error Handling

In a rush to get things rolled out, one of the things left half-done is adding enough error handling in scripts. Automation scripts that are not robust can cause major production issues, which could impact the credibility of DevOps efforts itself. A DevOps engineer should be aware of the following best practices in error handling and logging:

• The importance of error handling in automated scripts.

• Error handling in Bash.

• Error handling in Python.

• Logging errors in application and system logs.

Introduction

This article provides DIGIT Infra overview, Guidelines for Operational Excellence while DIGIT is deployed on SDC, NIC or any commercial clouds along with the recommendations and segregation of duties (SoD). It helps to plan the procurement and build the necessary capabilities to deploy and implement DIGIT.

In a shared control, the state program team/partners can consider these guidelines and must provide their own control implementation to the state’s cloud infrastructure and partners for a standard and smooth operational excellence.

Operational Recommendations

DIGIT strongly recommends Site reliability engineering (SRE) principles as a key means to bridge between development and operations by applying a software engineering mindset to system and IT administration topics. In general, an SRE team is responsible for availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning.

Monitoring Tools Recommendations: Commercial clouds like AWS, Azure and GCP offer sophisticated monitoring solutions across various infra levels like CloudWatch and StackDriver, in the absence of such managed services to monitor we can look at various best practices and tools listed below which helps debugging and troubleshooting efficiently.

• Network monitoring, Diagnosis, Assessment, Identify the latency ****

• Systems

• Application monitoring

• Web application monitoring

• Alert management, Alert Types

• Distributed Tracing

• Telemetry

Key Standard Operating Procedures (SOPs):

• Segregation of duties and responsibilities.

• SME and SPOCs for L1.5 support along with the SLAs defined.

• Ticketing system to manage incidents, converge and collaborate on various operational issues.

• Monitoring dashboards at various levels like Infrastructure, Network and applications.

• Transparency of monitoring data and collaboration between teams.

• Periodic remote sync up meetings, acceptance and attendance to the meeting.

• Ability to see stakeholders availability of calendar time to schedule meetings.

• Periodic (weekly, monthly) summary reports of the various infra, operations incident categories.

• Communication channels and synchronization on a regular basis and also upon critical issues, changes, upgrades, releases etc.

Segregation of Duties:

While DIGIT is deployed at state cloud infrastructure, it is essential to identify and distinguish the responsibilities between Infrastructure, Operations and Implementation partners. Identify these teams and assign SPOC, define responsibilities and Incident management is followed to visualize, track issues and manage dependencies between teams. Essentially these are monitored through dashboards and alerts are sent to the stakeholders proactively. eGov team can provide consultation and training on the need basis depending any of the below categories.

• State program team - Refers to the owner for the whole DIGIT implementation, application rollouts, capacity building. Responsible for identifying and synchronizing the operating mechanism between the below teams.

• Implementation partner - Refers to the DIGIT Implementation, application performance monitoring for errors, logs scrutiny, TPS on peak load, distributed tracing, DB queries analisis, etc.

• Operations team - this team could be an extension of the implementation team who is responsible for DIGIT deployments, configurations, CI/CD, change management, traffic monitoring and alerting, log monitoring and dashboard, application security, DB Backups, application uptime, etc.

• **State IT/Cloud team -**Refers to state infra team for the Infra, network architecture, LAN network speed, internet speed, OS Licensing and upgrade, patch, compute, memory, disk, firewall, IOPS, security, access, SSL, DNS, data backups/recovery, snapshots, capacity monitoring dashboard.

Skills Required to Set up, Operate and Maintain DIGIT on SDC:

Best practices for securing your Kubernetes cluster

Kubernetes has changed the way organizations deploy and run their applications, and it has created a significant shift in mindsets. While it has already gained a lot of popularity and more and more organizations are embracing the change, running Kubernetes in production requires care.

Although Kubernetes is open source and does it have its share of vulnerabilities, making the right architectural decision can prevent a disaster from happening.

You need to have a deep level of understanding of how Kubernetes works and how to enforce the best practices so that you can run a secure, highly available, production-ready Kubernetes cluster.

Although Kubernetes is a robust container orchestration platform, the sheer level of complexity with multiple moving parts overwhelms all administrators.

That is the reason why Kubernetes has a large attack surface, and, therefore, hardening of the cluster is an absolute must if you are to run Kubernetes in production.

There are a massive number of configurations in K8s, and while you can configure a few things correctly, the chances are that you might misconfigure a few things.

I will describe a few best practices that you can adopt if you are running Kubernetes in production. Let’s find out.

Use a Managed Kubernetes Service if Possible

If you are running your Kubernetes cluster in the cloud, consider using a managed Kubernetes cluster such as Google Kubernetes Engine or Azure Kubernetes Service.

A managed cluster comes with some level of hardening already in place, and, therefore, there are fewer chances to misconfigure things. A managed cluster also makes upgrades easy, and sometimes automatic. It helps you manage your cluster with ease and provides monitoring and alerting out of the box.

Upgrade Kubernetes Frequently

Since Kubernetes is open source, vulnerabilities appear quickly and security patches are released regularly. You need to ensure that your cluster is up to date with the latest security patches and for that, add an upgrade schedule in your standard operating procedure.

Having a CI/CD pipeline that runs periodically for executing rolling updates for your cluster is a plus. You would not need to check for upgrades manually, and rolling updates would cause minimal disruption and downtime; also, there would be fewer chances to make mistakes.

That would make upgrades less of a pain. If you are using a managed Kubernetes cluster, your cloud provider can cover this aspect for you.

Patch and Harden Your OS

It goes without saying that you should patch and harden the operating system of your Kubernetes nodes. This would ensure that an attacker would have the least attack surface possible.

You should upgrade your OS regularly and ensure that it is up to date.

Enforce RBAC

Kubernetes post version 1.6 has role-based access control (RBAC) enabled by default. Ensure that your cluster has this enabled.

You also need to ensure that legacy attribute-based access control (ABAC) is disabled. Enforcing RBAC gives you several advantages as you can now control who can access your cluster and ensure that the right people have the right set of permissions.

RBAC does not end with securing access to the cluster by Kubectl clients but also by pods running within the cluster, nodes, proxies, scheduler, and volume plugins.

Only provide the required access to service accounts and ensure that the API server authenticates and authorizes them every time they make a request.

Use TLS

Running your API server on plain HTTP in production is a terrible idea. It opens your cluster to a man in the middle attack and would open up multiple security holes.

Always use transport layer security (TLS) to ensure that communication between Kubectl clients and the API server is secure and encrypted.

Be aware of any non-TLS ports you expose for managing your cluster. Also ensure that internal clients such as pods running within the cluster, nodes, proxies, scheduler, and volume plugins use TLS to interact with the API server.

Segregate Resources in Namespaces

While it might be tempting to create all resources within your default namespace, it would give you tons of advantages if you use namespaces. Not only will it be able to segregate your resources in logical groups but it will also enable you to define security boundaries to resources in namespaces.

Namespaces logically behave as a separate cluster within Kubernetes. You might want to create namespaces based on teams, or based on the type of resources, projects, or customers depending on your use case.

After that, you can do clever stuff like defining resource quotas, limit ranges, user permissions, and RBAC on the namespace layer.

Avoid binding ClusterRoles to users and service accounts, instead provide them namespace roles so that users have access only to their namespace and do not unintentionally misconfigure someone else’s resources.

Cluster Role and Namespace Role Bindings

Control Pod to Pod Traffic

You can use Kubernetes network policies that work as firewalls within your cluster. That would ensure that an attacker who gained access to a pod (especially the ones exposed externally) would not be able to access other pods from it.

You can create Ingress and Egress rules to allow traffic from the desired source to the desired target and deny everything else.

Kubernetes Network Policy

Create Separate User Accounts

By default, when you boot your cluster through kubeadm, you get access to the kubernetes-admin config file which is the superuser for performing all activities within your cluster.

Do not share this file within your team, instead, create a separate user account for every user and only provide the right accesses to them. Bear in mind that Kubernetes does not maintain an internal user directory, and therefore, you need to ensure that you have the right solution in place to create and manage your users.

Once you create the user, you can generate a private key and a certificate signing request for the user, and Kubernetes would sign and generate a CA cert for the user.

You can then securely share the CA certificate with the user. The user can then use the certificate within kubectl to authenticate with the API server securely.

Configuring User Accounts

Follow the Principle of Least Privilege

You can provide granular access to user and service accounts with RBAC. Let us consider a typical organization where you can have multiple roles, such as:

1. Application developers — These need access only to a namespace and not the entire cluster. Ensure that you provide them with access only to deploy their applications and troubleshoot them only within their namespace. You might want application developers with access to spin only ClusterIP services and might wish to grant permissions only to network administrators to define ingresses for them.

2. Network administrators — You can provide network admins access to networking features such as ingresses, and privileges to spin up external services.

3. Cluster administrators — These are sysadmins whose main job is to administer the entire cluster. These are the only people that should have cluster-admin access and only the amount that is necessary for them to do their roles.

The above is not etched in stone, and you can have a different organization policy and different roles, but the only thing to keep in mind here is that you need to enforce the principle of least privilege.

That means that individuals and teams should have only the right amount of access they need to perform their job, nothing less and nothing more.

Frequently Rotate Infrastructure Credentials

It does not stop with just issuing separate user accounts and using TLS to authenticate with the API server. It is an absolute must that you frequently rotate and issue credentials to your users.

Set up an automated system that periodically revokes the old TLS certificates and issues new ones to your user. That helps as you don’t want attackers to get hold of a TLS cert or a token and then make use of it indefinitely.

A bootstrap token, for example, needs to be revoked as soon as you finish with your activity. You can also make use of a credential management system such as HashiCorp Vault which can issue you with credentials when you need them and revoke them when you finish with your work.

Use a Partitioned Approach to Secure Secrets

Imagine a scenario where an externally exposed web application is compromised, and someone has gained access to the pod. In that scenario, they would be able to access the secrets (such as private keys) and target the entire system.

The way to protect from this kind of attack is to have a sidecar container that stores the private key and responds to signing requests from the main container.

In case someone gets access to your login microservice, they would not be able to gain access to your private key, and therefore, it would not be a straightforward attack, giving you valuable time to protect yourself.

Partitioned Approach

Limit Resource Usage

The last thing you would want as a cluster-admin is a situation where a poorly written microservice code that has a memory leak can take over a cluster node causing the Kubernetes cluster to crash. That is an extremely important and generally ignored area.

You can add a resource limit and requests on the pod level as a developer or the namespace as an administrator. You can use resource quotas to limit the amount of CPU, memory, or persistent disk a namespace can allocate.

It can also allow you to limit the number of pods, volumes, or services you can spin within a namespace. You can also make use of limit ranges that provide you with a minimum and maximum size of resources every unit of the cluster within the namespace can request.

That will limit users from seeking an unusually large amount of resources such as memory and CPU.

Specifying a default resource limit and request on a namespace level is generally a good idea as developers aren’t perfect. If they forget to specify a limit, then the default limit and requests would protect you from resource overrun.

Protect Your ETCD Cluster Like a Treasure Vault

The ETCD datastore is the primary source of data for your Kubernetes cluster. That is where all cluster information and the expected configuration is stored.

If someone gains access to your ETCD database, all security measures will go down the drain. They will have full control of your cluster, and they can do what they want by modifying state in your ETCD datastore.

You should always ensure that only the API server can communicate with the ETCD datastore and only through TLS using a secure mutual auth. You can put your ETCD nodes behind a firewall and block all traffic except the ones originating from the API server.

Do not use the master ETCD for any other purpose but for managing your Kubernetes cluster and do not provide any other component access to the ETCD cluster.

Enable encryption of your secret data at rest. That is extremely important so that if someone gets access to your ETCD cluster, they should not be able to view your secrets by just doing a hex dump of your secrets.

Control Container Privileges

Containers run on nodes and therefore have some level of access to the host file system, however, the best way to reduce the attack surface is to architect your application in such a way that containers do not need to run as root.

Use pod security policies to restrict the pod to access HostPath volumes as that might result in getting access to the host filesystem. Administrators can use a restrictive pod policy so that anyone who gained access to one pod should not be able to access another pod from there.

Enable Auditing

Audit loggers are now a beta feature in Kubernetes, and I recommend you make use of it. That would help you troubleshoot and investigate what happened in case of an attack.

As a cluster-admin dealing with a security incident, the last thing you would want is that you are unaware of what exactly happened with your cluster and who has done what.

Conclusion

Remember that the above are just some general best practices and they are not exhaustive. You are free to adjust and make changes based on your use case and ways of working for your team.

Overview

This page discusses the infrastructure requirements for DIGIT services. It also explains why DIGIT services are containerised and deployed on Kubernetes.

Requirements

DIGIT Infra is abstracted to Kubernetes which is an open-source containers orchestration platform that helps in abstracting a variety of infra types that are being available across each state, like Physical, VMs, on-premises clouds(VMware, OpenStack, Nutanix, etc.), commercial clouds (Google, AWS, Azure, etc.), SDC and NIC into a standard infra type. Essentially it unifies various infra types into a standard and single type of infrastructure and thus DIGIT becomes multi-cloud supported, portable, extensible, high-performant and scalable containerized workloads and services. This facilitates both declarative configuration and automation. Kubernetes services, eco-system, support and tools are widely available.

The basic need to provision Kubernetes Cluster

Kubernetes as such is a set of components that designated jobs of scheduling, controlling, monitoring

Master Cluster

• 3 or more machines running one of:

  • Ubuntu 16.04+

  • Debian 9

  • CentOS 7

  • RHEL 7

  • Container Linux (tested with 1576.4.0)

• 4 GB or more of RAM per machine (any less will leave little room for your apps)

• 2 CPUs or more

User Cluster

• 3 or more machines running one of:

  • Ubuntu 16.04+

  • Debian 9

  • CentOS 7

  • RHEL 7

  • Container Linux (tested with 1576.4.0)

• 2 GB or more of RAM per machine (any less will leave little room for your apps)

• 2 CPUs or more

• Full network connectivity between all machines in the cluster (public or private network is fine)

• Unique hostname, MAC address, and product_uuid for every node. Click here for more details.

• Certain ports are open on your machines. See below for more details

• Swap disabled. You MUST disable swap in order for the Kubelet to work properly

Verify the MAC Address and product_uuid Are Unique for Every Node

• You can get the MAC address of the network interfaces using the command ip link or ifconfig -a

• The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid

It is very likely that hardware devices will have unique addresses, although some virtual machines may have identical values. Kubernetes uses these values to uniquely identify the nodes in the cluster. If these values are not unique to each node, the installation process may fail.

Check Network Adapters

If you have more than one network adapter, and your Kubernetes components are not reachable on the default route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter.

Check Required Ports

Master Cluster Master Node(s)

Worker Node(s)& User Cluster Worker Nodes

** Default port range for NodePort Services.

Any port numbers marked with * are overridable, so you will need to ensure any custom ports you provide are also open.

Complete Infra Specifications

Introduction

This document aims to put together all the items which will enable us to come up with a proper training plan for a partner team who will be working on the eDCR service used for the plan scrutiny.

Technical Pre-requisites

Below listed are the technical skillsets that are required to work on eDCR service. It is expected that the team planning on attending training is well versed with the mentioned technologies before they attend eGov training sessions.

Skillset for the Development team

• Java and REST APIS

• Postgres

• Maven

• Spring framework

• Basics of 2D CAD Drawings

• Git

• Postman

• YAML/JSON

Skill Set for the DevOps team

• Strong working knowledge of Linux, command, VM Instances, networking, storage

• The session, cache, and tokens handling (Redis-server)

• Understanding of VM types, Linux OS types, LoadBalancer, VPC, Subnets, Security Groups, Firewall, Routing, DNS

• Experience setting up CI like Jenkins and creating pipelines

• Artifactory - Nexus, verdaccio, DockerHub, etc

• Experience in setting up SSL certificates and renewal

• Gitops, Git branching, PR review process. Rules, Hooks, etc.

• JBoss Wildfly, Apache, Nginx, Redis and Postgres

Hardware prerequisites

Trainees are expected to have laptops/ desktops configured as mentioned below with all the software required to run the eDCR service application

• Laptop for hands-on training with 16GB RAM and OS preferably Ubuntu

• All developers need to have Git ids

• Install VSCode/IntelliJ/Eclipse

• Install Git

• Install JDK 8 update 112 or higher

• Install maven v3.2.x

• Install PostgreSQL v9.6

• Postman

• Install LibreCAD

• Application Server JBoss Wildfly v11.x

Software Assets

There are knowledge assets available on the Net for general items and eGov assets for DIGIT services. Here you can find references to each of the topics of importance. It is mandated the trainees do a self-study of all the software mentioned in the prerequisites using the reference materials shared.

Prepare Azure Environment

For provisioning Kubernetes clusters with the Azure cloud provider Kubermatic needs a service account with (at least) the Azure role Contributor. Please follow the following steps to create a matching service account.

Login to Azure and Get Basic Information

Login to Azure with Azure CLI az.

This command will open in your default browser a window where you can authenticate. After you succefully logged in get your subscription ID.

az account show --query id -o json

********-****-****-****-************

Get your Tenant ID

az account show --query tenantId -o json

********-****-****-****-************

create a new app with

az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/********-****-****-****-************"

Retrying role assignment creation: 1/36
Retrying role assignment creation: 2/36
Retrying role assignment creation: 3/36
{
  "appId": "********-****-****-****-************",
  "displayName": "azure-cli-2018-11-25-08-01-39",
  "name": "http://azure-cli-2018-11-25-08-01-39",
  "password": "********-****-****-****-************",
  "tenant": "********-****-****-****-************"
}

Enter provider credentials using the values from step “Prepare Azure Environment” into Kubermatic Dashboard:

• Client ID: Take the value of appId

• Client Secret: Take the value of password

• Tenant ID: your tenant ID

• Subscription ID: your subscription ID

What to know when deploying Kubernetes on SDC

Running Kubernetes on-premise gives a cloud-native experience or SDC becomes cloud-agnostic when it comes to the experience of Deploying DIGIT.

Whether States have their own on-premise data centre, have decided to forego the various managed cloud solutions, there are few things one should know when getting started with on-premise K8s.

One should be familiar with Kubernetes and one should know that the control plane consists of the Kube-apiserver, Kube-scheduler, Kube-controller-manager and an ETCD datastore. For managed cloud solutions like Google’s Kubernetes Engine (GKE) or Azure’s Kubernetes Service (AKS) it also includes the cloud-controller-manager. This is the component that connects the cluster to the external cloud services to provide networking, storage, authentication, and other feature support.

To successfully deploy a bespoke Kubernetes cluster and achieve a cloud-like experience on SDC, one need to replicate all the same features you get with a managed solution. At a high-level this means that we probably want to:

• Automate the deployment process

• Choose a networking solution

• Choose a storage solution

• Handle security and authentication

Let us look at each of these challenges individually, and we’ll try to provide enough of an overview to aid you in getting started.

Automating the deployment process

Using a tool like an ansible can make deploying Kubernetes clusters on-premise trivial.

When deciding to manage your own Kubernetes clusters, we need to set up a few proof-of-concept (PoC) clusters to learn how everything works, perform performance and conformance tests, and try out different configuration options.

After this phase, automating the deployment process is an important if not necessary step to ensure consistency across any clusters you build. For this, you have a few options, but the most popular are:

• ****kubeadm: a low-level tool that helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices

• kubespray: an ansible playbook that helps deploy production- ready clusters

If you already using ansible, kubespray is a great option otherwise we recommend writing automation around kubeadm using your preferred playbook tool after using it a few times. This will also increase your confidence and knowledge in the tooling surrounding Kubernetes.

Choosing a network solution

When designing clusters, choosing the right container networking interface (CNI) plugin can be the hardest part. This is because choosing a CNI that will work well with an existing network topology can be tough. Do you need BGP peering capabilities? Do you want an overlay network using vxlan? How close to bare-metal performance are you trying to get?

There are a lot of articles that compare the various CNI provider solutions (calico, weave, flannel, kube-router, etc.) that are must-reads like the benchmark results of Kubernetes network plugins article. We usually recommend Project Calico for its maturity, continued support, and large feature set or flannel for its simplicity.

For ingress traffic, you’ll need to pick a load-balancer solution. For a simple configuration, you can use MetalLB, but if you’re lucky enough to have F5 hardware load-balancers available we recommend checking out the K8s F5 BIG-IP Controller. The controller supports connecting your network plugin to the F5 either through either vxlan or BGP peering. This gives the controller full visibility into pod health and provides the best performance.

Choosing a storage solution

Kubernetes provides a number of included storage volume plugins. If you’re going on-premise you’ll probably want to use network-attached storage (NAS) option to avoid forcing pods to be pinned to specific nodes.

For a cloud-like experience, you’ll need to add a plugin to dynamically create persistent volume objects that match the user’s persistent volume claims. You can use dynamic provisioning to reclaim these volume objects after a resource has been deleted.

Pure Storage has a great example helm chart, the Pure Service Orchestrator (PSO), that provides smart provisioning although it only works for Pure Storage products.

Handle security and authentication

As anyone familiar with security knows, this is a rabbit-hole. You can always make your infrastructure more secure and should be investing in continual improvements.

Including different Kubernetes plugins can help build a secure, cloud-like experience for your users

When designing on-premise clusters you’ll have to decide where to draw the line. To really harden your cluster’s security you can add plugins like:

• istio: provides the underlying secure communication channel, and manages authentication, authorization, and encryption of service communication at scale

• gVisor: is a user-space kernel, written in Go, that implements a substantial portion of the Linux system surface

• vault: secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data

For user authentication, we recommend checking out guard which will integrate with an existing authentication provider. If you’re already using Github teams to then this could be a no-brainer.

Other Considerations

Hope this has given you a good idea of deploying, networking, storage, and security for you to take the leap into deploying your own on-premise Kubernetes clusters. Like we mentioned above, the team will want to build proof-of-concept clusters, run conformance and performance tests, and really become experts on Kubernetes if you’re going to be using it to run DIGIT on production.

We’ll leave you with a few other things the team should be thinking of:

• Externally backing up Kubernetes YAML, namespaces, and configuration files

• Running applications across clusters in an active-active configuration to allow for zero-downtime updates

• Running game days like deleting the CNI to measure and improve time-to-recovery

Introduction

This document aims to put together all the items which will enable us to come up with a proper training plan for a partner team that will be working on the DIGIT platform.

Technical Prerequisites

Below listed are the technical skill sets that are required to work on the DIGIT stack. It is expected the team planning on attending training is well versed with the mentioned technologies before they attend eGov training sessions.

Skillset for the Development team

• Open API Contract - Swagger2.0

• YAML/JSON

• Postman

• Postgres

• Java and REST APIS

• Basics of Elasticsearch

• Maven

• Springboot

• Kafka

• Zuul

• NodeJS, ReactJS

• WordPress

• PHP

Skill Set for the DevOps team

• Understanding of the microservice architecture.

• Experience of AWS, Azure, GCP, NIC Cloud.

• Strong working knowledge of Linux, command, VM Instances, networking, storage.

• To create Kubernetes cluster on AWS, Azure, GCP on NIC Cloud.

• Kubectl installation & commands (apply, get, edit, describe k8s objects)

• Terraform for infra-as-code for cluster or VM provisioning.

• Understanding of VM types, Linux OS types, LoadBalancer, VPC, Subnets, Security Groups, Firewall, Routing, DNS)

• Experience setting up CI like Jenkins and create pipelines.

• Deployment strategies - Rolling updates, Canary, Blue/Green.

• Scripting - Shell, Groovy, Python and GoLang.

• Experience in Baking Containers and Dockers.

• Artifactory - Nexus, Verdaccio, DockerHub, etc.

• Experience on Kubernetes ingress, setting up SSL certificates and renewal

• Understanding on Zuul gateway

• Gitops, Git branching, PR review process. Rules, Hooks, etc.

• Experience in Helm, packaging and deploying.

• JBoss Wildfly, Apache, Nginx, Redis and Postgres.

Hardware prerequisites

Trainees are expected to have laptops/ desktops configured as mentioned below with all the software required to run the DIGIT application

• Laptop for hands-on training with 16GB RAM and OS preferably Ubuntu

• All developers need to have Git ids

• Install VSCode/IntelliJ/Eclipse

• Install Git

• Install JDK 8 update 112 or higher

• Install maven v3.2.x

• Install PostgreSQL v9.6

• Install Elastic Search v2.4.x

• Postman

Software Assets

There are knowledge assets available in the Net for general items and eGov assets for DIGIT services. Here you can find references to each of the topics of importance. It is mandated the trainees do a self-study of all the software mentioned in the prerequisites using the reference materials shared.

Overview

This release offers the pre-payment and post-payment options to users while submitting the request for desludging operations along with multi-trip support.

Release Highlights

• Payment preference

• Multiple trips per service request

Release Features

Enhancements

Known Issues

• FSTPO inbox sort

Upcoming Release Features

• Better UX/UI for citizen application flow

• Dashboard with insights on Payment Preference, Gender, Request Status and Customer Rating

Reference Doc Links

MDMS Changes

Config Changes

Infra Changes

Service Build Updates

Download the file below to view the DIGIT Rollout Program Governance structure and process details.

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

This section discusses the supported cloud environment for DIGIT services. It provides information on where and how DIGIT is deployed. Further, it offers guidelines on estimating the infrastructural requirements for cloud support.

Supported Cloud List

• Google Cloud

• Azure

• AWS

• VSphere

• SDC

• NIC

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

Overview

This page discusses the provisioning of the Kubernetes cluster which is an abstracted infrastructure requirement for DIGIT to be deployed. Learn how to provision infra-as-code on AWS using terraform.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetInstanceProfile",
                "iam:ListInstanceProfiles"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:instance-profile/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:DeleteRolePolicy",
                "iam:GetRole",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:PassRole",
                "iam:PutRolePolicy"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:role/kubernetes-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:AddRoleToInstanceProfile",
                "iam:CreateInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:GetInstanceProfile",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:instance-profile/kubernetes-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteRule",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeregisterTargets",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyRule",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetRulePriorities",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:SetWebAcl",
                "sts:GetFederationToken"
            ],
            "Resource": "*"
        }
    ]
}

This section contains a list of documents elaborating on the key concepts aiding the deployment of the DIGIT platform.

All content on this page by is licensed under a .

Containerising applications and running them on Kubernetes doesn’t mean we can forget all about resource utilization. Our thought process may have changed because we can much more easily scale-out our application as demand increases, but many times we need to consider how our containers might fight with each other for resources. Resource Requests and Limits can be used to help stop the “noisy neighbour” problem in a Kubernetes Cluster.

Resource Requests

To put things simply, a resource request specifies the minimum amount of resources a container needs to successfully run. Thought of in another way, this is a guarantee from Kubernetes that you’ll always have this amount of either CPU or Memory allocated to the container.

Why would you worry about the minimum amount of resources guaranteed to a pod? Well, its to help prevent one container from using up all the node’s resources and starving the other containers from CPU or memory. For instance, if I had two containers on a node, one container could request 100% of that nodes processor. Meanwhile, the other container would likely not be working very well because the processor is being monopolized by its “noisy neighbour”.

What a resource request can do, is ensure that at least a small part of that processor’s time is reserved for both containers. This way if there is resource contention, each pod will have a guaranteed, minimum amount of resources in which to still function.

Resource Limits

As you might guess, a resource limit is the maximum amount of CPU or memory that can be used by a container. The limit represents the upper bounds of how much CPU or memory that a container within a pod can consume in a Kubernetes cluster, regardless of whether or not the cluster is under resource contention.

Limits prevent containers from taking up more resources on the cluster than you’re willing to let them.

Common Practices

As a general rule, all containers should have a request for memory and CPU before deploying to a cluster. This will ensure that if resources are running low, your container can still do the minimum amount of work to stay in a healthy state until those resources free up again (hopefully).

Limits are often used in conjunction with requests to create a “guaranteed pod”. This is where the request and limit are set to the same value. In that situation, the container will always have the same amount of CPU available to it, no more or less.

At this point, you may be thinking about adding a high “request” value to make sure you have plenty of resources available for your container. This might sound like a good idea, but have dramatic consequences for scheduling on the Kubernetes cluster. If you set a high CPU request, for example, 2 CPUs, then your pod will ONLY be able to be scheduled on Kubernetes nodes that have 2 full CPUs available that aren’t reserved by other pods’ requests. In the example below, the 2 vCPU pods couldn’t be scheduled on the cluster. However, if you were to lower the “request” amount to say 1 vCPU, it could.

Resource Requests and Limits – In Action

CPU Limit Example