This page provides information on how to deploy DIGIT services on Kubernetes, prepare deployment manifests for various services along with its configurations, secrets. etc. It also discusses the maintenance of environment-specific changes.

​All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.

This section contains architectural details about DIGIT deployment. It discusses the various activities in a sequence of steps to provision required infra and deploy DIGIT.

Sample Kubernetes Architecture

DIGIT Deployment Architecture

The CI/CD Flow

• Every code commit is well-reviewed and squash merge to branches through Pull Requests.

• Trigger the CI Pipeline that ensures code quality, vulnerability assessments, CI tests before building the artefacts.

• Artefact is version controlled based on Semantic versioning based on the nature of the change.

• After successful CI, Jenkins bakes the Docker Images with the versioned artefacts and pushes the baked docker image to Docker Registry.

• Deployment Pipeline pulls the built Image and pushes to the corresponding Env.

Deployment Scripts

• As all the DIGIT services that are containerized and deployed on Kubernetes, we need to prepare deployment manifests. The same can be found here.

• DIGIT has built helm charts to using the standard helm approach to ease managing the service-specific configs, customisations, switch/toggle, secrets, etc.

• Golang base Deployment script that reads the values from the helm charts template and deploys into the cluster.

• Each env will have one master yaml template that will have the definition of all the services to be deployed, their dependencies like Config, Env, Secrets, DB Credentials, Persistent Volumes, Manifest, Routing Rules, etc..

Overview

Once the cluster is ready and healthy you can start deploying backbones services.

Deploy configuration and deployment in the following Services Lists

1. Backbone (Redis, ZooKeeper-v2, Kafka-v2,elasticsearch-data-v1, elasticsearch-client-v1, elasticsearch-master-v1)

2. Gateway (Zuul, nginx-ingress-controller)

Pre-requisites

• Understanding of VM Instances, LoadBalancers, SecurityGroups/Firewalls, nginx, DB Instance, Data Volumes.

• Experience of Kubernetes, Docker, Jenkins, helm, golang, Infra-as-code.

Deploy configuration and deployment backbone services:

1. Clone the git repo https://github.com/egovernments/eGov-infraOps . Copy existing dev.yaml and dev-secrets.yaml with new environment name (eg..yaml and-secrets.yaml)

2. Modify the global domain and set namespaces create to true

1. Modify the below-mentioned changes for each backbone services:

   Eg. For Kafka-v2 If you are using AWS as cloud provider, change the respective volume id’s and zone’s

   (You will get the volume id’s and zone details from either remote state bucket or from AWS portal)

Eg. Kafka-v2 If you are using Azure cloud provider, change the diskName and diskUri

(You will get the volume id’s and zone details from either remote state bucket or from Azure portal)

Eg. Kafka-v2 If you are using ISCSI , change the targetPortal and iqn.

1. Deploy the backbone services using go command

cd /eGov-infraOps/egov-deployergo run main.go deploy -e dev -p -c 'kafka-v2,redis,zookeeper-v2,elasticsearch-data-v1,elasticsearch-master-v1,playground,cert-manager,kafka-connect,kafka-connect-restart-tasks,kibana-v1,nginx-ingress'

Modify the “dev” environment name with your respective environment name.

Flags:

• e --- Environment name

• p --- Print the manifest

• c --- Enable Cluster Configs

• Check the Status of pods

kubectl get pods --all-namespaces

​

Overview

In Kubernetes, an Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. You configure access by creating a collection of rules that define which inbound connections reach which services.

This lets you consolidate your routing rules into a single resource. For example, you might want to send requests to example.com/api/v1/ to an api-v1 service, and requests to example.com/api/v2/ to the api-v2 service. With an Ingress, you can easily set this up without creating a bunch of LoadBalancers or exposing each service on the Node.

An API object that manages external access to the services in a cluster, typically HTTP.

Ingress may provide load balancing, SSL termination and name-based virtual hosting.

Terminology

For clarity, this guide defines the following terms:

• Node: A worker machine in Kubernetes, part of a cluster.

• Cluster: A set of Nodes that run containerized applications managed by Kubernetes. For this example, and in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.

• Edge router: A router that enforces the firewall policy for your cluster. This could be a gateway managed by a cloud provider or a physical piece of hardware.

• Cluster network: A set of links, logical or physical, that facilitate communication within a cluster according to the Kubernetes networking model.

• Service: A Kubernetes Service that identifies a set of Pods using label selectors. Unless mentioned otherwise, Services are assumed to have virtual IPs only routable within the cluster network.

What is Ingress?

​Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

  internet        | [ Ingress ]   --|-----|--   [ Services ]

An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.

An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

Prerequisites

You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.

You may need to deploy an Ingress controller such as ingress-nginx. You can choose from a number of Ingress controllers.

Ideally, all Ingress controllers should fit the reference specification. In reality, the various Ingress controllers operate slightly differently.

The Ingress resource

An Ingress resource example:

apiVersion: extensions/v1beta1kind: Ingressmetadata:  annotations:    kubernetes.io/ingress.class: nginx  name:  service1  namespace: egovspec:  rules:  - host: foo.bar.com    http:      paths:      - backend:          serviceName:  service1          servicePort: 8080        path: /foo  tls:  - hosts:    - foo.bar.com    secretName: foo.bar.com-tls-certs

​

As with all other Kubernetes resources, an Ingress needs apiVersion, kind, and metadata fields. The name of an Ingress object must be a valid DNS subdomain name. For general information about working with config files, see deploying applications, configuring containers, managing resources. Ingress frequently uses annotations to configure some options depending on the Ingress controller, an example of which is the rewrite-target annotation. Different Ingress controller support different annotations. Review the documentation for your choice of Ingress controller to learn which annotations are supported.

The Ingress spec has all the information needed to configure a load balancer or proxy server. Most importantly, it contains a list of rules matched against all incoming requests. Ingress resource only supports rules for directing HTTP(S) traffic.

Ingress rules

Each HTTP rule contains the following information:

• An optional host. In this example, no host is specified, so the rule applies to all inbound HTTP traffic through the IP address specified. If a host is provided (for example, foo.bar.com), the rules apply to that host.

• A list of paths (for example, /testpath), each of which has an associated backend defined with a service.name and a service.port.name or service.port.number. Both the host and path must match the content of an incoming request before the load balancer directs traffic to the referenced Service.

• A backend is a combination of Service and port names as described in the Service doc or a custom resource backend by way of a CRD. HTTP (and HTTPS) requests to the Ingress that matches the host and path of the rule are sent to the listed backend.

A defaultBackend is often configured in an Ingress controller to service any requests that do not match a path in the spec.

• Learn about the Ingress API​

• Learn about Cert-manager​