Kubernetes Guide

What is Kubernetes?

Kubernetes is an open source cluster management framework initially released by Google. It schedules Docker containers (workloads) to run across a fleet of nodes. The nodes can be virtual machines from cloud providers like Amazon Web Services or even physical machines.

Kubernetes and Glints

As a software engineer at Glints, for the most part, you do not have to interact directly with Kubernetes or understand how it works. However, it may be helpful to know how to perform certain tasks, especially for debugging and troubleshooting.

This guide also does not intend to outline what has already been included in Kubernetes' own excellent documentation. Kubernetes is flexible and therefore many companies will have different ways of setting things up. Therefore, we will mostly focus on things that are specific to our own setup.

Accessing the Cluster

Rancher, our UI to manage the Kubernetes cluster is located here. If you do not have access, ping a senior developer to add you to our GitHub organization. After authenticating with your GitHub account, select the development cluster.

For most operations, you will rely on Kubernetes' official command line tool, kubectl. You may download kubectl on your platform of choice by following the instructions here.

kubectl uses a YAML-based configuration file (usually called kubeconfig) to determine which cluster to access and what credentials to use. In the Rancher interface, you can download the kubeconfig used to access our cluster. You may also use the web-based kubectl for quick operations.

Once configured, try out:

kubectl cluster-info

Feature Branch Organization

We currently use Kubernetes to implement our "review apps" system, in which each feature branch in our codebase has a corresponding URL. For example, in the Employers project, if a branch called feature/omnisearch is pushed to GitLab, an environment at https://employers-feature-omnisearch.dev.glints.com will be created.

On Kubernetes, each feature branch that is pushed gets its own namespace. You can see the list of namespaces currently in the cluster via the Rancher UI, or by using the following command:

kubectl get ns

Namespaces allow logical isolation of resources on the cluster. In addition, a pod in a namespace can access other services in the same namespace by name via DNS. By default, kubectl uses the default namespace.

To see all resources in a namespace, use the following command:

kubectl --namespace=*namespace* get all

For a feature branch like feature/omnisearch, this would be:

kubectl --namespace=feature/omnisearch get all
kubectl -n=feature/omnisearch get all          # Shorter version

If you find typing the namespace tedious, you can set a namespace as default:

kubectl config set-context $(kubectl config current-context) --namespace=feature/omnisearch

Connecting to Services

By default, pods started within the cluster run with the ClusterIP networking configuration. This means that they are assigned an IP that is internal within the cluster, and is not accessible from the Internet.

To expose an endpoint to the Internet, an ingress resource can be used to route traffic to a specific service within the cluster. Our configuration files for projects such as api and dst already do this by default.

However, if you want to access a service that is only available within the cluster, you may use kubectl port-forward. For example, to access a Postgres pod for the feature/omnisearch branch:

kubectl -n=feature-omnisearch get pods  # Retrieve the list of pods
kubectl -n=feature-omnisearch port-forward postgres-0 8888:5432

The above command will make Postgres' port 5432 accessible via localhost:8888. You may then use standard tools to access, for example:

psql --host=localhost --port=8888 --username=glints ...

Debugging and Troubleshooting

Occasionally, you might want to find out why your service isn't working as expected. To do this, you can get logs:

kubectl -n=feature-omnisearch get pods  # Get the name of the pod
kubectl -n=feature-omnisearch logs -f api-worker-6dc747d794-5xmhb

You can also get shell access:

kubectl -n=feature-omnisearch exec -it api-worker-6dc747d794-5xmhb sh

Changing Configuration

In some cases, you might want to change the environment variables passed to your service. One example of this is updating which API instance to use for the frontend projects. By default, both dst and employers point to api.staging.glints.com and not the API instance within their namespace.

Configuration is generally stored in Kubernetes using ConfigMaps, which are simple key-value pairs that are injected into pods as environment variables.

To get a list of ConfigMaps:

kubectl -n=feature-omnisearch get configmaps

To see the data within the ConfigMap:

kubectl -n=feature-omnisearch get configmap dst -o yaml
kubectl -n=feature-omnisearch get configmap dst -o json  # Use JSON format

To edit the ConfigMap:

kubectl -n=feature-omnisearch edit configmap dst

More Reading

A wealth of information on Kubernetes can be found on the official documentation.

Current State of Migration

As of date of publishing, feature branches have been migrated from Rancher 1.x (Cattle) to Rancher 2.x (Kubernetes). Migration of the staging and production environments will take place once the feature branch environment is stable and when we gain more familiarity with Kubernetes.