1. Infra Setup

Prerequisites:

• Linux distribution running in a VM or bare metal

  • Install curl, wget git, and tar (if they're not already installed):

    • sudo apt-get install curl git wget tar

  • Install Docker

  • Install kubectl on Linux

  • Open terminal and Install k3d(v4.4.8) on Linux using the below command

    • wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v4.4.8 bash

  OSX or Mac

  • Docker Desktop local Kubernetes cluster enabled

  • Install kubectl on Mac

  • Install k3d on Mac, on terminal use curl command.

  • Homebrew (Homebrew is available for MacOS) using the below command to install curl.

  Copy

  brew install curl
  curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG=v4.4.8 bash

• Windows 10 or above

  • You should have admin access to follow the steps

  • Docker Desktop for windows need to be installed ( make sure wsl is installed to run Docker Desktop )

  • Install kubectl on Windows

  • Install Chocolatey package manager for windows

  • Install GitBash as an alternative command prompt that allows most of the Linux commands on windows.

  • Install k3d(v4.4.8)

Create Lightweight Kubernetes Cluster

Ensure that the docker is running and you have the admin privileges for the device. Run the following in the terminal/command prompt.

Note: Make sure the k3d is already installed as part of the pre-requisites.

Start with the setup of the lightweight Kubernetes cluster on your local machine/VM. Execute all the instructions as admin/root/sudoer to give the installer full access to provision the required system resources/folders etc.

1. First, Create "Kube" directory in any desired place (ensure you use the right dir path, if it is different from the example) and change permission. This will be used as k3d cluster persistent storage to store metadata and container logs.

   • If Linux/Mac

     1. Copy

        cd ~
        mkdir kube
        chmod 777 kube
        cd kube
        pwd  #copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path

   • If Windows

     1. Copy

        cd D:\
        mkdir kube
        wsl chmod -R 777 kube
        cd kube
        pwd  #copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path

2. Create a k3d cluster with a single master node and 2 agents (Worker Nodes) and mount the above created directory (for data persistence). Here's where it stores all the metadata and persists the logs of the workloads.

   • Copy

     k3d cluster create --k3s-server-arg "--no-deploy=traefik" --agents 2 -v "/home/<user_name>/kube:/kube@agent[0,1]" -v "/home/<user_name>/kube:/kube@server[0]" --port "80:80@loadbalancer"

   *NOTE: Update "/home/<user_name>/kube" this path in above cmd with your respective absolute path.

3. When cluster creation is successful, Get the kubeconfig file, which will allow you to connect the to the cluster.

   • Copy

     k3d kubeconfig get k3s-default > myk3dconfig

   • Copy

     kubectl config use-context k3d-k3s-default --kubeconfig=myk3dconfig

4. Verify the Cluster creation by running the following commands from your local machine where the kubectl is installed. It gives you the sample output as below if everything works fine.

   1. Copy

      kubectl cluster-info

      OutPut

      Copy

      Kubernetes control plane is running at https://0.0.0.0:33931
      CoreDNS is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
      Metrics-server is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

   2. Copy

      k3d cluster list

      OutPut:

      Copy

      NAME          SERVERS   AGENTS   LOADBALANCER
      k3s-default   1/1       2/2      true

   3. Copy

      kubectl get nodes

      OutPut:

      Copy

      NAME                       STATUS   ROLES                  AGE     VERSION
      k3d-k3s-default-agent-0    Ready    <none>                 3d18h   v1.21.1+k3s1
      k3d-k3s-default-agent-1    Ready    <none>                 3d18h   v1.21.1+k3s1
      k3d-k3s-default-server-0   Ready    control-plane,master   3d18h   v1.21.1+k3s1

   4. Copy

      kubectl top nodes

      OutPut:

      Copy

      W0625 07:56:24.588781   12810 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
      NAME                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
      k3d-k3s-default-agent-0    547m         6%     1505Mi          9%        
      k3d-k3s-default-agent-1    40m          0%     2175Mi          13%       
      k3d-k3s-default-server-0   59m          0%     2286Mi          14%  

Pre-read:

Know what is terraform: https://youtu.be/h970ZBgKINg

Prerequisites

1. Install terraform for the Infra-as-code (IaC) to provision cloud resources as code and with desired resource graph and also it helps to destroy the cluster at one go. The desired terraform version is 0.14.10.

   1. If you already have a different version of the terraform version running install tfswitch that would allow you to have multiple terraform versions in the same machine and toggle between the desired versions.

2. Setup your AWS account

3. If you are not using the eGov AWS account. In that case, you have to open 80, 22, and 6550 ports in a default security group

1. Launching VM/Instance using Terraform

1. Clone the following DIGIT Devops GitRepo, you may need to install git and then run git clone it to your machine.

   • Copy

     git clone -b quickstart https://github.com/egovernments/DIGIT-DevOps 

2. After cloning the repo CD into the folder DIGIT-DevOps and type the "code ." command that will open the visual editor and opens all the files from the repo DIGIT-DevOps

3. Copy

   cd DIGIT-DevOps/infra-as-code/terraform/quickstart-aws-ec2
   code .

4. Generate ssh key

5. The following main.tf will create ec2 instance.

provider "aws" {
  region = "${var.region}"
}
module "ssh_key" {
  source             = "../modules/instance/aws-ec2"
  key_name           =  "${var.key_name}"
  public_key         =  "${var.public_key}"
} 

resource "aws_instance" "digit-quickstart-vm" {
  ami                    = "${var.ami_name_value}"
  instance_type          = "${var.instance_type}"
  key_name               = module.ssh_key.ssh_key_name
  monitoring             =  false
  associate_public_ip_address = true
  availability_zone      =  "ap-south-1b"

  tags = {
    Name = "${var.tag}"
  }
}

5. Update variables.tf

variable "region" {
  default = "ap-south-1"
}

variable "ami_name_value" {
  default = "ami-0bb9e2d19522c61d4"
}

variable "instance_type" {
  default = "c5.2xlarge"
}

## The VM capacity may depend on the H/W requirement of the number of DIGIT modules that you may try for the POC
## Eg: 


variable "tag" {
    default = "digit-quickstart-vm" ## change tag name eg. digit-quickstart-vm_your-name_name
}

## change ssh public_key with your public ssh key
variable "public_key" {
  default = "ssh-rsa <asfas>/Gy6w0PPSnnfl/AWXO7ckFtEXQbdz9Y15zeUFKgUsbklXxmC6D37BkPGu+IjCZSOttPV+PRM0Dnf0jQLvMV0UhEkguD9ALC5xikqNlFyPH5bGetWDxtLbn61tnoOIYG6lXAdk2Oe35yWWt3ZgcccWtYuRwDo0ofBwY9jWOkEcCefDyYg+S7h1VzNsbB9DsFv0vPcaxHcZK8bLdyhnz1+9rXy/flbiS5kE0O97aZ4zm4wAmqiivN2wWhUez18k2Mcs= demo@demo" 
  description = "ssh key"
}


variable "key_name" {
  default = "digit-quickstart-vm"  ## change ssh key_name eg.digit-quickstart-vm-your_name
  description = "ssh key name"
}

6. Export AWS Profile.

export AWS_PROFILE=digit-quickstart-poc

6. Terraform Execution:

cd DIGIT-DevOps/infra-as-code/terraform/quickstart-aws-ec2

terraform init
terraform plan
terraform apply

7. The terraform apply command displays the results of creating the resource defined in your configuration files.

Note: Copy the "Public IP" from the outputs.

module.ssh_key.aws_key_pair.ssh_key: Creating...
module.ssh_key.aws_key_pair.ssh_key: Creation complete after 0s [id=digit-quickstart-vm]
aws_instance.digit-quickstart-poc: Creating...
aws_instance.digit-quickstart-poc: Still creating... [10s elapsed]
aws_instance.digit-quickstart-poc: Creation complete after 12s [id=i-05764e3b780d836dd]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

public_ip = "<some_public_ip>"

8. terraform out provided you with vm/instance public IP use same to login.

ssh -i <path_to_ssh_private_key> ubuntu@<your_vm_public_ip>
NOTE: replace IP <your_vm_public_ip> with your respective public IP

After logging into the newly provisioned AWS ec2 instance perform the below steps

2. Create a lightweight Kubernetes Cluster

The VM Has most of the above mentioned pre-requisites, you can directly execute the below commands, incase you are trying on any other VM of your choice, ensure that you install all the mentioned softwares/tools and you have the admin access to your machine.

As a first step, Let's setup the lightweight kubernetes cluster on your VM. You must execute all the instructions as admin/root/sudoer to give the installer full access to provision the required system resources/folders etc.

• Create "Kube" directory in any desired place (ensure you use the right dir path, if it is different from the example) and change permission accordingly. This will be used as k3d cluster persistent storage to store all the metadata and container logs.

  1. Copy

     sudo mkdir ~/kube
     sudo chmod -R 777 ~/kube
     cd ~/kube
     pwd  
     
     #copy the path you get here. Provide an absolute path to below k3d cmd, by replacing "/home/<user_name>/kube" with your directory path
     eg: /home/<your_user_name>/kube

• Create a k3d cluster with a single master node and 2 agents (Worker Nodes) and mount the above created directory (for data persistence). Here's where it stores all the metadata and persists the logs of the workloads.

Notes:

1. Update "/home/<your_user_name>/kube" this path in below cmd with your respective absolute path

2. To route the traffic to the above created cluster to be able to connect to the cluster from your local machine/laptop using kubeconfig file

3. Update <VM_Private_IP> with VM Private IP, use below command on VM terminal to get private IP.

   Copy

   ip a | grep "inet " | grep -v  "127.0.0.1" | awk -F " " '{print $2}'|awk -F "/" '{print $1}'|head -n1   

4. Update <VM_Public_IP> with VM's Public IP which you used to ssh into VM.

• Copy

  sudo k3d cluster create \
  --api-port <VM_Private_IP>:6550 \
  --k3s-server-arg "--no-deploy=traefik" \
  --k3s-server-arg "--tls-san=<VM_Public_IP>" \
  --agents 2 \
  -v "/home/<your_user_name>/kube:/kube@agent[0,1]" \
  -v "/home/<your_user_name>/kube:/kube@server[0]" \
  --port "80:80@loadbalancer" 
  

• When cluster creation is successful, Get the kubeconfig file, which will allow you to connect the to the cluster.

  1. sudo k3d kubeconfig get k3s-default > myk3dconfig

  2. sudo kubectl config use-context k3d-k3s-default --kubeconfig=myk3dconfig

• Verify the Cluster creation by running the following commands from your AWS vm where the kubectl is installed. It gives you the sample output as below if everything works fine.

  1. sudo kubectl cluster-info

     OutPut

     Copy

     Kubernetes control plane is running at https://0.0.0.0:33931
     CoreDNS is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
     Metrics-server is running at https://0.0.0.0:33931/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

  2. sudo k3d cluster list

     OutPut:

     Copy

     NAME          SERVERS   AGENTS   LOADBALANCER
     k3s-default   1/1       2/2      true

  3. sudo kubectl get nodes

     OutPut:

     Copy

     NAME                       STATUS   ROLES                  AGE     VERSION
     k3d-k3s-default-agent-0    Ready    <none>                 3d18h   v1.21.1+k3s1
     k3d-k3s-default-agent-1    Ready    <none>                 3d18h   v1.21.1+k3s1
     k3d-k3s-default-server-0   Ready    control-plane,master   3d18h   v1.21.1+k3s1

  4. sudo kubectl top nodes

     OutPut:

     Copy

     W0625 07:56:24.588781   12810 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
     NAME                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
     k3d-k3s-default-agent-0    547m         6%     1505Mi          9%        
     k3d-k3s-default-agent-1    40m          0%     2175Mi          13%       
     k3d-k3s-default-server-0   59m          0%     2286Mi          14%  

To Access the k3s cluster from local

1. Copy myk3dconfig to your local machine/laptop

2. Replace the clusters.cluster.server Private IP with Public Ip from your myk3dconfig.

For reference observe the below kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 
    server: https://<VM_Private_IP>:6550 //replace <VM_Private_IP> with <VM_Public_IP>
  name: k3d-k3s-default
contexts:
- context:
    cluster: k3d-k3s-default
    user: admin@k3d-k3s-default
  name: k3d-k3s-default
current-context: k3d-k3s-default
kind: Config
preferences: {}
users:
- name: admin@k3d-k3s-default
  user:
    client-certificate-data: 
    client-key-data:

3. Setup kubeconfig

export KUBECONFIG=<path-to-your-kube_config>

4. To check the nodes from local run the below command

kubectl get nodes --kubeconfig=<path-to-your-kube_config>

Deploy DIGIT

Now you can start with the Deployment.

Destroy VM/Instance

Once you are done with the Quickstart setup, Destroy the terraform resources.

cd DIGIT-DevOps/infra-as-code/terraform/digit-quickstart-aws
terraform destroy