Introduction to Kubernetes

Kubernetes is a powerful system, developed by Google, for managing containerized applications in a clustered environment. It aims to provide better ways of managing related, distributed components across varied infrastructure.

In this guide, we'll discuss some of Kubernetes' basic concepts. We will talk about the architecture of the system, the problems it solves, and the model that it uses to handle containerized deployments and scaling.

This article assumes some prior knowledge about modern cluster technology. We will be discussing how it relates to systems like CoreOS.

If you are not familiar with CoreOS, it may be helpful to review some basic information about the CoreOS system in order to understand the types of environments that Kubernetes is meant to be deployed on.

Kubernetes, at its basic level, is a system for managing containerized applications across a cluster of nodes. In many ways, Kubernetes was designed to address the disconnect between the way that modern, clustered infrastructure is designed, and some of the assumptions that most applications and services have about their environments.

Most clustering technologies strive to provide a uniform platform for application deployment. The user should not have to care much about where work is scheduled. The unit of work presented to the user is at the "service" level and can be accomplished by any of the member nodes.

However, in many cases, it does matter what the underlying infrastructure looks like. When scaling an app out, an administrator cares that the various instances of a service are not all being assigned to the same host.

On the other side of things, many distributed applications build with scaling in mind are actually made up of smaller component services. These services must be scheduled on the same host as related components if they are going to be configured in a trivial way. This becomes even more important when they rely on specific networking conditions in order to communicate appropriately.

While it is possible with most clustering software to make these types of scheduling decisions, operating at the level of individual services is not ideal. Applications comprised of different services should still be managed as a single application in most cases. Kubernetes provides a layer over the infrastructure to allow for this type of management.

Infrastructure-level systems like CoreOS strive to create a uniform environment where each host is disposable and interchangeable. Kubernetes, on the other hand, operates with a certain level of host specialization.

The controlling unit in a Kubernetes cluster is called the master server. It serves as the main management contact point for administrators, and it also provides many cluster-wide systems for the relatively dumb worker nodes.

The master server runs a number of unique services that are used to manage the cluster's workload and direct communications across the system. Below, we will cover the components that are specific to the master server.

One of the fundamental components that Kubernetes needs to function is a globally available configuration store. The etcd project, developed by the CoreOS team, is a lightweight, distributed key-value store that can be distributed across multiple nodes.

Kubernetes uses etcd to store configuration data that can be used by each of the nodes in the cluster. This can be used for service discovery and represents the state of the cluster that each component can reference to configure or reconfigure themselves. By providing a simple HTTP/JSON API, the interface for setting or retrieving values is very straight forward.

The implementation of etcd on a Kubernetes cluster is a bit more flexible than CoreOS. If you choose to, you can turn off etcd on every server but your master, so long as you point your Kubernetes components at the master server's instance. Because the master server must be up for Kubernetes to function, the value of distributing the store is less fundamental.

One of the most important services that the master server runs is an API server. This is the main management point of the entire cluster, as it allows a user to configure many of Kubernetes' workloads and organizational units. It also is responsible for making sure that the etcd store and the service details of deployed containers are in agreement.

The API server implements a RESTful interface, which means that many different tools and libraries can readily communicate with it. A client called kubecfg is packaged along with the server-side tools and can be used from a local computer or by connecting to the master server.

The controller manager service is used to handle the replication processes defined by replication tasks. The details of these operations are written to etcd, where the controller manager watches for changes.

When a change is seen, the controller manager reads the new information and implements the replication procedure that fulfills the desired state. This can involve scaling the application group up or down.

The process that actually assigns workloads to specific nodes in the cluster is the scheduler. This is used to read in a service's operating requirements, analyze the current infrastructure environment, and place the work on an acceptable node or nodes.

The scheduler is responsible for tracking resource utilization on each host to make sure that workloads are not scheduled in excess of the available resources. The scheduler must know the total resources available on each server, as well as the resources allocated to existing workloads assigned on each server.

In Kubernetes, servers that perform work are known as minions. Minion servers have a few requirements that are necessary to communicate with the master, configure the networking for containers, and run the actual workloads assigned to them.

The first requirement of each individual minion server is docker. The docker service is used to run encapsulated application containers in a relatively isolated but lightweight operating environment. Each unit of work is, at its basic level, implemented as a series containers that must be deployed.

One key assumption that Kubernetes makes is that a dedicated subnet is available to each minion server. This is not the case with many standard clustered deployments. For instance, with CoreOS, a separate networking fabric called flannel is needed for this purpose. Docker must be configured to use this so that it can expose ports in the correct fashion.

The main contact point for each minion with the cluster group is through a small service called kubelet. This service is responsible for relaying information to and from the master server, as well as interacting with the etcd store to read configuration details or write new values.

The kubelet service communicates with the master server to receive commands and work. Work is received in the form of a "manifest" which defines the workload and the operating parameters. Thekubelet process then assumes responsibility for maintaining the state of the work on the minion server.

In order to deal with individual host subnetting and in order to make services available to external parties, a small proxy service is run on each minion server. This process forwards requests to the correct containers, can do primitive load balancing, and is generally responsible for making sure the networking environment is predictable and accessible, but isolated.

While containers are the used to deploy applications, the workloads that define each type of work are specific to Kubernetes. We will go over the different types of "work" that can be assigned below.

A pod is the basic unit that Kubernetes deals with. Containers themselves are not assigned to hosts. Instead, closely related containers are grouped together in a pod. A pod generally represents one or more containers that should be controlled as a single "application".

This association leads all of the involved containers to be scheduled on the same host. They are managed as a unit and they share an environment. This means that they can share volumes and IP space, and can be deployed and scaled as a single application. You can and should generally think of pods as a single virtual computer in order to best conceptualize how the resources and scheduling should work.

The general design of pods usually consists of the main container that satisfies the general purpose of the pod, and optionally some helper containers that facilitate related tasks. These are applications that benefit from being run and managed in their own container, but are heavily tied to the main application.

Horizontal scaling is generally discouraged with pods because there are other units more suited for the task.

We have been using the term "service" throughout this guide in a very loose fashion, but Kubernetes actually has a very specific definition for the word when describing work units. A service, when described this way, is a unit that acts as a basic load balancer and ambassador for other containers.

This allows you to deploy a service unit that is aware of all of the backend containers to pass traffic to. External applications only need to worry about a single access point, but benefit from a scalable backend or at least a backend that can be swapped out when necessary.

Services are an interface to a group of containers so that consumers do not have to worry about anything beyond a single access location. By deploying a service, you easily gain discover-ability and can simplify your container designs.

A more complex version of a pod is a replicated pod. These are handled by a type of work unit known as areplication controller.

A replication controller is a framework for defining pods that are meant to be horizontally scaled. The work unit is, in essence, a nested unit. A template is provided, which is basically a complete pod definition. This is wrapped with additional details about the replication work that should be done.

The replication controller is delegated responsibility over maintaining a desired number of copies. This means that if a container temporarily goes down, the replication controller might start up another container. If the first container comes back online, the controller will kill off one of the containers.

A Kubernetes organizational concept outside of the work-based units is labeling. A label is basically an arbitrary tag that can be placed on the above work units to mark them as a part of a group. These can then be selected for management purposes and action targeting.

Labels are fundamental to how both services and replication controllers function. To get a list of backend servers that a service should pass traffic to, it selects containers based on labels that have been given to them.

Similarly, replication controllers give all of the containers spawned from their templates the same label. This makes it easy for the controller to monitor each instance. The controller or the administrator can manage all of the instances as a group, regardless of how many containers have been spawned.

Labels are given as key-value pairs. Each unit can have more than one label, but each unit can only have one entry for each key. You can stick with giving pods a "name" key as a general purpose identifier, or you can classify them by various criteria such as development stage, public accessibility, etc.

In many cases you'll want to assign many labels for fine-grained control. You can then select based on a single or combined label requirements.

Kubernetes is an exciting project that implements many functional improvements on top of clustered infrastructure. While other technologies do a great job at handling the clustering aspects, Kubernetes aims to offer a better management system.