Prerequisites

1. GitHub Organization account

2. Fork the belo repo's to your GitHub Organization account 1. https://github.com/egovernments/DIGIT-DevOps and 2. https://github.com/egovernments/CIOps

3. AWS KMS

4. Go lang (version 1.13.X)

5. SOPS

6. GitHub user

7. Docker Hub account

8. ​AWS account with the admin access to provision EKS Service, you can always subscribe to free AWS account to learn the basics and try, but there is a limit to what is offered as free, for this demo you need to have a commercial subscription to the EKS service, if you want to try out for a day or two, it might cost you about Rs 500 - 1000. (Note: Post the Demo, for the internal folks, eGov will provide a 2-3 hrs time bound access to eGov's AWS account based on the request and available number of slots per day)

9. Install kubectl on your local machine that helps you interact with the kubernetes cluster

10. Install Helm that helps you package the services along with the configurations, envs, secrets, etc into a kubernetes manifests

11. Install terraform version (0.14.10) for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go.

12. ​Install AWS CLI on your local machine so that you can use aws cli commands to provision and manage the cloud resources on your account.

13. Install AWS IAM Authenticator that helps you authenticate your connection from your local machine so that you should be able to deploy DIGIT services.

14. Use the AWS IAM User credentials provided for the Terraform (Infra-as-code) to connect with your AWS account and provision the cloud resources.

    ​

    1. You'll get a Secret Access Key and Access Key ID. Save them safely.

    2. Open the terminal and Run the following command you have already installed the AWS CLI and you have the credentials saved. (Provide the credentials and you can leave the region and output format as blank)

    Copy

    aws configure --profile cicd-infra-account 
    
    AWS Access Key ID []:<Your access key>
    AWS Secret Access Key []:<Your secret key>
    Default region name []: ap-south-1
    Default output format []: text

    1. The above will create the following file In your machine as /Users/.aws/credentials

    Copy

    [cicd-infra-account] 
    aws_access_key_id=*********** 
    aws_secret_access_key=****************************

CI/CD Cluster Setup

Terraform helps you build a graph of all your resources, and parallelizes the creation and modification of any non-dependent resources. Because of this, Terraform builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure.

Before we provision the cloud resources, we need to understand and be sure about what resources need to be provisioned by terraform to deploy CI/CD.

The following is the resource graph that we are going to provision using terraform in a standard way so that every time and for every env, it'll have the same infra.

• EKS Control Plane (Kubernetes Master)

• Work node group (VMs with the estimated number of vCPUs, Memory)

• EBS Volumes (Persistent Volumes)

• VPCs (Private network)

• Users to access, deploy and read-only

Understand the Resource Graph in Terraform script:

• Ideally, one would write the terraform script from the scratch using this doc.

• Here we have already written the terraform script that provisions the production-grade DIGIT Infra and can be customized with the specified configuration.

• Let's Clone the DIGIT-DevOps GitHub repo where the terraform script to provision EKS cluster is available and below is the structure of the files.

git clone --branch release https://github.com/egovernments/DIGIT-DevOps.git
cd DIGIT-DevOps/infra-as-code/terraform


└── modules
    ├── kubernetes
    │   └── aws
    │       ├── eks-cluster
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       ├── network
    │       │   ├── main.tf
    │       │   ├── outputs.tf
    │       │   └── variables.tf
    │       └── workers
    │           ├── main.tf
    │           ├── outputs.tf
    │           └── variables.tf
    └── storage
        └── aws
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

In here, you will find the main.tf under each of the modules that has the provisioning definition for resources like EKS cluster, and Storage, etc. All these are modularized and reacts as per the customized options provided.

Example:

• VPC Resources:

  • VPC

  • Subnets

  • Internet Gateway

  • Route Table

• EKS Cluster Resources:

  • IAM Role to allow EKS service to manage other AWS services

  • EC2 Security Group to allow networking traffic with EKS cluster

  • EKS Cluster

• EKS Worker Nodes Resources:

  • IAM role allowing Kubernetes actions to access other AWS services

  • EC2 Security Group to allow networking traffic

  • Data source to fetch latest EKS worker AMI

  • AutoScaling Launch Configuration to configure worker instances

  • AutoScaling Group to launch worker instances

• Storage Module

  • Configuration in this directory creates EBS volume and attaches it together.

• The following main.tf with create s3 bucket to store all the state of the execution to keep track

• Copy

  DIGIT-DevOps/Infra-as-code/terraform/egov-cicd/remote-state
  
  [**main.tf**](https://github.com/egovernments/DIGIT-DevOps/blob/release/infra-as-code/terraform/egov-cicd/remote-state/main.tf)\*\*\*\*

provider "aws" {
  region = "ap-south-1"
}

#This is a bucket name that you can name as you wish
resource "aws_s3_bucket" "terraform_state" {
  bucket = "try-cicd-workshop-yourname" 

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

#This is a bucket name that you can name as you wish
resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "try-cicd-workshop-yourname"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

1. The following main.tf contains the detailed resource definitions that need to be provisioned, please have a look at it.

   Dir: DIGIT-DevOps/Infra-as-code/terraform/egov-cicd

2. main.tf

terraform {
  backend "s3" {
    bucket = "try-cicd-workshop-yourname"
    key = "terraform"
    region = "ap-south-1"
  }
}

module "network" {
  source             = "../modules/kubernetes/aws/network"
  vpc_cidr_block     = "${var.vpc_cidr_block}"
  cluster_name       = "${var.cluster_name}"
  availability_zones = "${var.network_availability_zones}"
}

module "iam_user_deployer" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-deployer"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_admin" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-admin"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "egovterraform" has uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

module "iam_user_user" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-user"

  name          = "${var.cluster_name}-kube-user"
  force_destroy = true  
  create_iam_user_login_profile = false
  create_iam_access_key         = true

  # User "test" has uploaded his public key here - https://keybase.io/test/pgp_keys.asc
  pgp_key = "${var.iam_keybase_user}"
}

data "aws_eks_cluster" "cluster" {
  name = "${module.eks.cluster_id}"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "${module.eks.cluster_id}"
}
provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.cluster.token}"
  load_config_file       = false
  version                = "~> 1.11"
}

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "${var.cluster_name}"
  cluster_version = "${var.kubernetes_version}"
  subnets         = "${concat(module.network.private_subnets, module.network.public_subnets)}"

  tags = "${
    map(
      "kubernetes.io/cluster/${var.cluster_name}", "owned",
      "KubernetesCluster", "${var.cluster_name}"
    )
  }"

  vpc_id = "${module.network.vpc_id}"

  worker_groups_launch_template = [
    {
      name                    = "spot"
      subnets                 = "${slice(module.network.private_subnets, 0, length(var.availability_zones))}"
      override_instance_types = "${var.override_instance_types}"
      asg_max_size            = "${var.number_of_worker_nodes}"
      asg_desired_capacity    = "${var.number_of_worker_nodes}"
      kubelet_extra_args      = "--node-labels=node.kubernetes.io/lifecycle=spot"
      spot_allocation_strategy= "lowest-price"
      spot_max_price          = "${var.spot_max_price}"
      spot_instance_pools     = 1
      cpu_credits             = "standard"
    },
  ]
  
  map_users    = [
    {
      userarn  = "${module.iam_user_deployer.this_iam_user_arn}"
      username = "${module.iam_user_deployer.this_iam_user_name}"
      groups   = ["system:masters"]
    },
    {
      userarn  = "${module.iam_user_admin.this_iam_user_arn}"
      username = "${module.iam_user_admin.this_iam_user_name}"
      groups   = ["global-readonly", "digit-user"]
    },
    {
      userarn  = "${module.iam_user_user.this_iam_user_arn}"
      username = "${module.iam_user_user.this_iam_user_name}"
      groups   = ["global-readonly"]
    },    
  ]
}

module "jenkins" {

  source = "../modules/storage/aws"
  storage_count = 1
  environment = "${var.cluster_name}"
  disk_prefix = "jenkins-home"
  availability_zones = "${var.availability_zones}"
  storage_sku = "gp2"
  disk_size_gb = "20"
  
}

Custom variables/configurations:

You can define your configurations in variables.tf and provide the env specific cloud requirements so that using the same terraform template you can customize the configurations.

├── egov-cicd
│   ├── main.tf 
│   ├── outputs.tf
│   ├── providers.tf
│   ├── remote-state
│   │   └── main.tf
│   └── variables.tf

Following are the values that you need to mention in the following files, the blank ones will be prompted for inputs while execution.

****variables.tf

#
# Variables Configuration
#

variable "cluster_name" {
  default = "<Desired Cluster name>"  #eg: my-digit-cicd
}

variable "vpc_cidr_block" {
  default = "192.168.0.0/16"
}

variable "network_availability_zones" {
  default = ["ap-south-1b", "ap-south-1a"]
}

variable "availability_zones" {
  default = ["ap-south-1b"]
}

variable "kubernetes_version" {
  default = "1.18"
}

variable "instance_type" {
  default = "t3a.xlarge"
}

variable "override_instance_types" {
  default = ["t3.xlarge", "r5ad.xlarge", "r5a.xlarge", "t3a.xlarge"]
}

variable "number_of_worker_nodes" {
  default = "1"
}

variable "spot_max_price" {
  default = "0.0538"
}

variable "ssh_key_name" {
  default = "egov-cicd"
}
variable "iam_keybase_user" {
  default = "keybase:egovterraform"
}

Important: Create your own keybase key before you run the terraform

• Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private key in your machine, upload the public key into the keybase account that you have just created, and give a name to it and ensure that you mention that in your terraform. This allows to encrypt all the sensitive information.

  • Example user keybase user in eGov case is "egovterraform" needs to be created and has to uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc

  • you can use this portal to Decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.

Run terraform

Now that we know what the terraform script does, the resources graph that it provisions and what custom values should be given with respect to your env.

Let's begin to run the terraform scripts to provision infra required to Deploy DIGIT on AWS.

1. First CD into the following directory and run the following command 1-by-1 and watch the output closely.

cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd/remote-state
terraform init
terraform plan
terraform apply


cd DIGIT-DevOps/infra-as-code/terraform/egov-cicd
terraform init
terraform plan
terraform apply

Upon Successful execution following resources gets created which can be verified by the command "terraform output"

• s3 bucket: to store terraform state.

• Network: VPC, security groups.

• IAM users auth: using keybase to create admin, deployer, the user. Use this URL https://keybase.io/ to create your own PGP key, this will create both public and private key in your machine, upload the public key into the keybase account that you have just created, and give a name to it and ensure that you mention that in your terraform. This allows to encrypt all the sensitive information.

  • Example user keybase user in eGov case is "egovterraform" needs to be created and has to uploaded his public key here - https://keybase.io/egovterraform/pgp_keys.asc

  • you can use this portal to Decrypt your secret key. To decrypt PGP Message, Upload the PGP Message, PGP Private Key and Passphrase.

• EKS cluster: with master(s) & worker node(s).

• Storage(s): for es-master, es-data-v1, es-master-infra, es-data-infra-v1, zookeeper, kafka, kafka-infra.

• Use this link to get the kubeconfig from EKS to get the kubeconfig file and being able to connect to the cluster from your local machine so that you should be able to deploy DIGIT services to the cluster.

aws sts get-caller-identity

# Run the below command and give the respective region-code and the cluster name
aws eks --region <region-code> update-kubeconfig --name <cluster_name>

1. Finally, Verify that you are able to connect to the cluster by running the following command

kubectl config use-context <your cluster name>

kubectl get nodes

NAME                                             STATUS AGE   VERSION               OS-Image           
ip-192-168-xx-1.ap-south-1.compute.internal   Ready  45d   v1.18.10-eks-bac369   Amazon Linux 2   

Whola! All set and now you can go Deploy Jenkins...

2. Jenkins Deployment

Post infra setup (Kubernetes Cluster), We start with deploying the Jenkins and kaniko-cache-warmer.

Prerequisites:

• Sub Domain to expose CI/CD URL

• GitHub Oauth App

• GitHub User ssh key

• With GitHub user generate a personal read only access token

• Docker hub account details (username and password)

• SSL Certificate for the sub-domain

Prepare an <ci.yaml> master config file and <ci-secrets.yaml>, you can name this file as you wish which will have the following configurations.

• credentials, secrets (You need to encrypt using sops and create a ci-secret.yaml separately)

• Check and Update ci-secrets.yaml ****details (like githuh Oauth app clientId and clientSecret, GitHub user details gitReadSshPrivateKey and gitReadAccessToken etc..)

• To create Jenkins namespace mark this flag true

• Add your env's kubconfigs under kubConfigs like https://github.com/egovernments/DIGIT-DevOps/blob/release/config-as-code/environments/ci-demo-secrets.yaml#L50

• KubeConfig env's name and deploymentJobs name from ci.yaml should be the same

• Update the CIOps and DIGIT-DevOps repo name with your forked repo name and provide read-only access to github user to those repo's.

• SSL Certificate for the sub-domain

cd DIGIT-DevOps/tree/release/deploy-as-code

kubectl config use-context <your cluster name>
go run main.go deploy -c -e ci 'jenkins,kaniko-cache-warmer,nginx-ingress,cert-manager'

You have launch the Jenkins, Same you can access through your sub-domain which you configured in ci.yaml

Overview

Since there are many DIGIT services and the development code is part of various git repos, you need to understand the concept of cicd-as-service which is open sourced. This page also guides you through the process of creating a CI/CD pipeline.

To integrate any new service/app to the CI/CD below is the starting point:

Once the desired service is ready for the integration: decide the service name, type of service, whether DB migration is required or not. While you commit the source code of the service to the git repository, the following file should be added with the relevant details which are mentioned as below:

Build-config.yml –It is present under the build directory in the repository

https://raw.githubusercontent.com/egovernments/DIGIT-OSS/master/build/build-config.yml

This file contains the below details which are used for creating the automated Jenkins pipeline job for your newly created service.

config:
 -   name: < Name of the job, foo/bar would create job named bar inside folder foo >
     build:
     - work-dir: < Working directory of the app to be built >
       dockerfile: < Path to the dockerfile, optional, assumes dockerfile in working directory if not provided >
       image-name: < Docker image name  >

While integrating a new service/app, the above content needs to be added in the build-config.yml file of that app repository. For example: If we are on-boarding a new service called egov-test, then the build-config.yml should be added as mentioned below.

config:  
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test      
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test

If a job requires multiple images to be created (DB Migration) then it should be added as below,

config:   
- name: builds/DIGIT-OSS/core-services/egov-test     
  build:     
  - work-dir: egov-test       
    dockerfile: build/maven/Dockerfile       
    image-name: egov-test     
  - work-dir: egov-test/src/main/resources/db       
    dockerfile: egov-test/src/main/resources/db/Dockerfile       
    image-name: egov-test-db

Note - If a new repository is created then the build-config.yml should be created under the build folder and then the config values are added to it.

The git repository URL is then added to the Job Builder parameters

When the Jenkins Job => job builder is executed the CI Pipeline gets created automatically based on the above details in build-config.yml. Eg: egov-test job will be created under builds/DIGIT-OSS/core-services folder in Jenkins because the “build-config was edited under core-services” And it should be the “master” branch only. Once the pipeline job is created, it can be executed for any feature branch with build parameters (Specifying which branch to be built – master or any feature branch).

As a result of the pipeline execution, the respective app/service docker image will be built and pushed to the Docker repository.

On repo provide read-only access to GitHub user (created while ci/cd deployment )

****

Continuous Integration (CI)

The Jenkins CI pipeline is configured and managed 'as code'.

• Job Builder – Job Builder is a Generic Jenkins job which creates the Jenkins pipeline automatically which are then used to build the application, create the docker image of it and push the image to docker repository. The Job Builder job requires the git repository URL as a parameter. It clones the respective git repository and reaads the build/build-config.yml file for each git repository and uses it to create the service build job.

check and ‌add your repo ssh url in ci.yaml​‌‌

If git repository ssh URL is available build the Job-Builder Job

If git repository URL is not available please check and add the same team.

**Continuous Deployment (CD)**‌

The services deployed and managed on a Kubernetes cluster in cloud platforms like AWS, Azure, GCP, OpenStack, etc. Here, we use helm charts to manage and generate the Kubernetes manifest files and use them for further deployment to respective Kubernetes cluster. Each service is created as charts which will have the below-mentioned files in it.

billing-service/   
# Directory – name of the service/appChart.yaml         
# A YAML file containing information about the chartLICENSE            
# OPTIONAL: A plain text file containing the license for the chartREADME.md          # OPTIONAL: A human-readable README filevalues.yaml        # The default configuration values for this charttemplates/         # A directory of templates that, when combined with values, will generate valid Kubernetes manifest files.

To deploy a new service, we need to create the helm chart for it. The chart should be created under the charts/helm directory in Digit-DeOps repository.

Github repository     
https://github.com/egovernments/DIGIT-DevOps/tree/release/deploy-as-code/helm/charts
We have an automatic helm chart generator utility which needs to be installed on the local machine, the utility will prompt for user inputs about the newly developed service( app specifications) for creating the helm chart. The requested chart with the configuration values (created based on the inputs provided) will be created for the user.

‌ _Name of the service? test-service Application Type? NA Kubernetes health checks to be enabled? Yes Flyway DB migration container necessary? No Expose service to the internet? Yes Route through API gateway [zuul] No Context path? hello_‌

The generated chart will have the following files.

create Chart.yaml 
create values.yaml
create templates/deployment.yaml
create templates/service.yaml
create templates/ingress.yaml

This chart can also be modified further based on user requirements.

The Deployment of manifests to the Kubernetes cluster is made very simple and easy. We have Jenkins Jobs for each state and environment-specific. We need to provide the image name or the service name in the respective Jenkins deployment job.

Enter a caption for this image (optional)

Enter a caption for this image (optional)

‌The deployment Jenkins job internally performs the following operations,‌

• Reads the image name or the service name given and finds the chart that is specific to it.

• Generates the Kubernetes manifests files from the chart using helm template engine.

• Execute the deployment manifest with the specified docker image(s) to the Kubernetes cluster.