Kubernetes in Production

So now you have figured out Containerized Workloads, Containers, Pods, Nodes, Deployments, Services, Ingress, Control Plane, etc., etc., etc. You are excited to see that you can keep killing a running app and it just comes back. Wow! indestructible apps.

Next, you configure Jenkins, a Docker Registry, and a configuration tool like Kustomize (https://kustomize.io/) to enable CI/CD within your cluster. But wait, your developer would need to use skaffold (https://skaffold.dev/) to run this whole infra on his/her machine.

You have also figured out that you would use your RDBMS as a Kubernetes pod and use Persistent Volumes to store the data. Not to forget that <your favorite noSql> is a stateful clustered application that will now run on a clustered service. Your applications - sorry services are stateless containers and storages are stateful container.

And finally voila! we are Kubernetizing.

Ok, that sounds like it's a very complicated thing - which Kubernetes or k8 as it's called - actually is. But it gives you really useful capabilities that will be very difficult to build on your own. For example:

Portability: Cluster can be deployed on any kind of environment single cloud, across cloud environments or on-premise

Availability: Improves availability of all the components running on the cluster

Elasticity: You can startup additional instances of an application or shutdown as you require

Networking: Lookup and Registry of all the applications on the cluster and outside of cluster is configuration-based and network or infra agnostic

Storage Management: All the data volumes that containers require can be configured once and then the cluster ensures its mounting, access control, etc

Deployment Topology: Based on deployment descriptors the cluster will decide which node a certain app should run on or not run on

Application Configurations: Environment properties and environment-specific application properties can be defined as deployment configurations. Complete separation of concerns

But of course, there is no free lunch! And for a framework that provides for such varied capabilities, it is bound to be complex. Remember running Kubernetes cluster in production is a different matter altogether - unless you are using it as a managed service from one of the cloud providers (azure, aws, gcp). So now let's go through some pointers on what one should be preparing for when they plan to use Kubernetes at an enterprise level. The pointers are in no specific order but important ones that need to be considered.

1. Security

At multiple levels. First things first, the k8 cluster needs to be secured. Access control to the Kubernetes Control Plane needs to be fine-tuned. Enable RBAC for all users accessing the cluster. Access to Pods and other containers within the cluster. Then of course security of all the containers running on the cluster. For e.g. running containers as non-root users, etc. Ensuring that there are no security vulnerabilities from the base images you use for your containers. Upgrades to the base images also need to be handled to avoid known security risks or bugs.

2. Backup

The Kubernetes cluster is dynamic and distributed so it needs to maintain its state. It does this using etcd. So you need to backup your etcd storage regularly if you want to plan for catastrophic failures and business continuity. Also since infra-as-code is the paradigm, you need to ensure all your configuration is defined as code and stored in a versioning system like git. These configurations include application deployment descriptors, resource definitions (DB, routers, etc). The idea is, in a fresh environment the state of your current cluster should be re-creatable using automated scripts rather than a sys-admin setting up everything manually.

3. Storage

There are two ways to deploy storage. First, you configure the tuning parameters for your DB image and deploy them on the k8 cluster. Or you can just use a base pre-configured image available from the DB provider. The second approach works well if you are using a licensed storage like Oracle DB. But for most free ones like Postgres or MySql you should go ahead and tune your DB. Both have their pros and cons. No right or wrong here

4. Monitoring

This is one of the most important cluster-wide features you will need. You will need to monitor applications for throughput, memory, IO at each container level. Host infra like network with latency, disk usage, IO, etc. And since the watcher needs to be watched a monitoring dashboard for the k8 cluster and docker.

OS logs, network logs, application logs, DB logs, security logs, etc - every component and piece of code generates logs. You need to aggregate them in a central store and continuously run analytics to watch for any problems. It is humanely impossible to watch so many parameters on different machines so automated alerts from the monitoring system are required

5. DevOps

Following the DevOps approach to development will require learning and adoption from the development team. The way applications are built, deployed and how they run in a k8 cluster needs to be understood by everyone involved in the application development and deployment. The developer has to be aware of how resources like databases and files will be made available to the application. Even application-specific properties are defined at the cluster level and not part of the application binary. Also, deployment strategies need to be decided to ensure optimal resource utilization and at the same time ensuring enough will be available to each application

6. Testing and Debugging

Writing distributed stateless software is a very tough task. Debugging and tracing it is even more difficult. How do you trace a transaction that goes through multiple services and does multiple operations/parallel operations? Log aggregations help you here but it's still a complex task. And if you plan to do parallel deployments of different versions of your code, you really need to be aware of the consequences and how to trace/follow request paths in such an environment.

7. Versatility

As mentioned earlier Kubernetes covers application deployment, networking, os interactions, storage, and many other things. This makes it difficult for one team or person to completely manage a full cluster. Thus it is essential that people from development and deployment, networking, and other IT departments including production support are involved with the maintenance of this infrastructure.

Each of the above topics can be expanded to an article of its own. But these are good starting points before you go ahead and decide on deploying a Kubernetes cluster for your client or organization. It Will be good to know your experiences, challenges you faced while deploying a k8 cluster on your system and how you overcame them.

Happy Coding!!