Node-local load balancing#
For clusters that don't have an externally managed load balancer for the k0s
control plane, there is another option to get a highly available control plane,
at least from within the cluster. K0s calls this "node-local load balancing". In
contrast to an externally managed load balancer, node-local load balancing takes
place exclusively on the worker nodes. It does not contribute to making the
control plane highly available to the outside world (e.g. humans interacting
with the cluster using management tools such as Lens or
kubectl), but rather makes the cluster itself internally resilient to
controller node outages.
Technical functionality#
The k0s worker process manages a load balancer on each worker node's loopback interface and configures the relevant components to use that load balancer. This allows for requests from worker components to the control plane to be distributed among all currently available controller nodes, rather than being directed to the controller node that has been used to join a particular worker into the cluster. This improves the reliability and fault tolerance of the cluster in case a controller node becomes unhealthy.
Envoy is the only load balancer that is supported so far. Please note that Envoy is not available on ARMv7, so node-local load balancing is currently unavailable on that platform.
Enabling in a cluster#
In order to use node-local load balancing, the cluster needs to comply with the following:
- The cluster doesn't use an externally managed load balancer, i.e. the cluster
configuration doesn't specify a non-empty
spec.api.externalAddress. - K0s isn't running as a single node, i.e. it isn't
started using the
--singleflag. - The cluster should have multiple controller nodes. Node-local load balancing also works with a single controller node, but is only useful in conjunction with a highly available control plane.
Add the following to the cluster configuration (k0s.yaml):
spec:
network:
nodeLocalLoadBalancing:
enabled: true
type: EnvoyProxy
Or alternatively, if using k0sctl, add the following to
the k0sctl configuration (k0sctl.yaml):
spec:
k0s:
config:
spec:
network:
nodeLocalLoadBalancing:
enabled: true
type: EnvoyProxy
All newly added worker nodes will then use node-local load balancing. The k0s worker process on worker nodes that are already running must be restarted for the new configuration to take effect.
Full example using k0sctl#
The following example shows a full k0sctl configuration file featuring three
controllers and two workers with node-local load balancing enabled:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s-cluster
spec:
k0s:
version: v1.31.3+k0s.0
config:
spec:
network:
nodeLocalLoadBalancing:
enabled: true
type: EnvoyProxy
hosts:
- role: controller
ssh:
address: 10.81.146.254
keyPath: k0s-ssh-private-key.pem
port: 22
user: k0s
- role: controller
ssh:
address: 10.81.146.184
keyPath: k0s-ssh-private-key.pem
port: 22
user: k0s
- role: controller
ssh:
address: 10.81.146.113
keyPath: k0s-ssh-private-key.pem
port: 22
user: k0s
- role: worker
ssh:
address: 10.81.146.198
keyPath: k0s-ssh-private-key.pem
port: 22
user: k0s
- role: worker
ssh:
address: 10.81.146.51
keyPath: k0s-ssh-private-key.pem
port: 22
user: k0s
Save the above configuration into a file called k0sctl.yaml and apply it in
order to bootstrap the cluster:
$ k0sctl apply
⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███ ███ ███
⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███ ███ ███
⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███ ███ ███
⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████ ███ ██████████
k0sctl 0.17.2 Copyright 2023, k0sctl authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
level=info msg="==> Running phase: Connect to hosts"
level=info msg="[ssh] 10.81.146.254:22: connected"
level=info msg="[ssh] 10.81.146.184:22: connected"
level=info msg="[ssh] 10.81.146.113:22: connected"
level=info msg="[ssh] 10.81.146.51:22: connected"
level=info msg="[ssh] 10.81.146.198:22: connected"
level=info msg="==> Running phase: Detect host operating systems"
level=info msg="[ssh] 10.81.146.254:22: is running Alpine Linux v3.17"
level=info msg="[ssh] 10.81.146.113:22: is running Alpine Linux v3.17"
level=info msg="[ssh] 10.81.146.184:22: is running Alpine Linux v3.17"
level=info msg="[ssh] 10.81.146.198:22: is running Alpine Linux v3.17"
level=info msg="[ssh] 10.81.146.51:22: is running Alpine Linux v3.17"
level=info msg="==> Running phase: Acquire exclusive host lock"
level=info msg="==> Running phase: Prepare hosts"
level=info msg="[ssh] 10.81.146.113:22: installing packages (curl)"
level=info msg="[ssh] 10.81.146.198:22: installing packages (curl, iptables)"
level=info msg="[ssh] 10.81.146.254:22: installing packages (curl)"
level=info msg="[ssh] 10.81.146.51:22: installing packages (curl, iptables)"
level=info msg="[ssh] 10.81.146.184:22: installing packages (curl)"
level=info msg="==> Running phase: Gather host facts"
level=info msg="[ssh] 10.81.146.184:22: using k0s-controller-1 as hostname"
level=info msg="[ssh] 10.81.146.51:22: using k0s-worker-1 as hostname"
level=info msg="[ssh] 10.81.146.198:22: using k0s-worker-0 as hostname"
level=info msg="[ssh] 10.81.146.113:22: using k0s-controller-2 as hostname"
level=info msg="[ssh] 10.81.146.254:22: using k0s-controller-0 as hostname"
level=info msg="[ssh] 10.81.146.184:22: discovered eth0 as private interface"
level=info msg="[ssh] 10.81.146.51:22: discovered eth0 as private interface"
level=info msg="[ssh] 10.81.146.198:22: discovered eth0 as private interface"
level=info msg="[ssh] 10.81.146.113:22: discovered eth0 as private interface"
level=info msg="[ssh] 10.81.146.254:22: discovered eth0 as private interface"
level=info msg="==> Running phase: Download k0s binaries to local host"
level=info msg="==> Running phase: Validate hosts"
level=info msg="==> Running phase: Gather k0s facts"
level=info msg="==> Running phase: Validate facts"
level=info msg="==> Running phase: Upload k0s binaries to hosts"
level=info msg="[ssh] 10.81.146.254:22: uploading k0s binary from /home/k0sctl/.cache/k0sctl/k0s/linux/amd64/k0s-v1.31.3+k0s.0"
level=info msg="[ssh] 10.81.146.113:22: uploading k0s binary from /home/k0sctl/.cache/k0sctl/k0s/linux/amd64/k0s-v1.31.3+k0s.0"
level=info msg="[ssh] 10.81.146.51:22: uploading k0s binary from /home/k0sctl/.cache/k0sctl/k0s/linux/amd64/k0s-v1.31.3+k0s.0"
level=info msg="[ssh] 10.81.146.198:22: uploading k0s binary from /home/k0sctl/.cache/k0sctl/k0s/linux/amd64/k0s-v1.31.3+k0s.0"
level=info msg="[ssh] 10.81.146.184:22: uploading k0s binary from /home/k0sctl/.cache/k0sctl/k0s/linux/amd64/k0s-v1.31.3+k0s.0"
level=info msg="==> Running phase: Configure k0s"
level=info msg="[ssh] 10.81.146.254:22: validating configuration"
level=info msg="[ssh] 10.81.146.184:22: validating configuration"
level=info msg="[ssh] 10.81.146.113:22: validating configuration"
level=info msg="[ssh] 10.81.146.113:22: configuration was changed"
level=info msg="[ssh] 10.81.146.184:22: configuration was changed"
level=info msg="[ssh] 10.81.146.254:22: configuration was changed"
level=info msg="==> Running phase: Initialize the k0s cluster"
level=info msg="[ssh] 10.81.146.254:22: installing k0s controller"
level=info msg="[ssh] 10.81.146.254:22: waiting for the k0s service to start"
level=info msg="[ssh] 10.81.146.254:22: waiting for kubernetes api to respond"
level=info msg="==> Running phase: Install controllers"
level=info msg="[ssh] 10.81.146.254:22: generating token"
level=info msg="[ssh] 10.81.146.184:22: writing join token"
level=info msg="[ssh] 10.81.146.184:22: installing k0s controller"
level=info msg="[ssh] 10.81.146.184:22: starting service"
level=info msg="[ssh] 10.81.146.184:22: waiting for the k0s service to start"
level=info msg="[ssh] 10.81.146.184:22: waiting for kubernetes api to respond"
level=info msg="[ssh] 10.81.146.254:22: generating token"
level=info msg="[ssh] 10.81.146.113:22: writing join token"
level=info msg="[ssh] 10.81.146.113:22: installing k0s controller"
level=info msg="[ssh] 10.81.146.113:22: starting service"
level=info msg="[ssh] 10.81.146.113:22: waiting for the k0s service to start"
level=info msg="[ssh] 10.81.146.113:22: waiting for kubernetes api to respond"
level=info msg="==> Running phase: Install workers"
level=info msg="[ssh] 10.81.146.51:22: validating api connection to https://10.81.146.254:6443"
level=info msg="[ssh] 10.81.146.198:22: validating api connection to https://10.81.146.254:6443"
level=info msg="[ssh] 10.81.146.254:22: generating token"
level=info msg="[ssh] 10.81.146.198:22: writing join token"
level=info msg="[ssh] 10.81.146.51:22: writing join token"
level=info msg="[ssh] 10.81.146.198:22: installing k0s worker"
level=info msg="[ssh] 10.81.146.51:22: installing k0s worker"
level=info msg="[ssh] 10.81.146.198:22: starting service"
level=info msg="[ssh] 10.81.146.51:22: starting service"
level=info msg="[ssh] 10.81.146.198:22: waiting for node to become ready"
level=info msg="[ssh] 10.81.146.51:22: waiting for node to become ready"
level=info msg="==> Running phase: Release exclusive host lock"
level=info msg="==> Running phase: Disconnect from hosts"
level=info msg="==> Finished in 3m30s"
level=info msg="k0s cluster version v1.31.3+k0s.0 is now installed"
level=info msg="Tip: To access the cluster you can now fetch the admin kubeconfig using:"
level=info msg=" k0sctl kubeconfig"
The cluster with the two nodes should be available by now. Setup the kubeconfig file in order to interact with it:
k0sctl kubeconfig > k0s-kubeconfig
export KUBECONFIG=$(pwd)/k0s-kubeconfig
The three controllers are available and provide API Server endpoints:
$ kubectl -n kube-node-lease get \
lease/k0s-ctrl-k0s-controller-0 \
lease/k0s-ctrl-k0s-controller-1 \
lease/k0s-ctrl-k0s-controller-2 \
lease/k0s-endpoint-reconciler
NAME HOLDER AGE
k0s-ctrl-k0s-controller-0 9ec2b221890e5ed6f4cc70377bfe809fef5be541a2774dc5de81db7acb2786f1 2m37s
k0s-ctrl-k0s-controller-1 fe45284924abb1bfce674e5a9aa8d647f17c81e53bbab17cf28288f13d5e8f97 2m18s
k0s-ctrl-k0s-controller-2 5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d 2m9s
k0s-endpoint-reconciler 9ec2b221890e5ed6f4cc70377bfe809fef5be541a2774dc5de81db7acb2786f1 2m37s
$ kubectl -n default get endpoints
NAME ENDPOINTS AGE
kubernetes 10.81.146.113:6443,10.81.146.184:6443,10.81.146.254:6443 2m49s
The first controller is the current k0s leader. The two worker nodes can be listed, too:
$ kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k0s-worker-0 Ready <none> 2m16s v1.31.3+k0s 10.81.146.198 <none> Alpine Linux v3.17 5.15.83-0-virt containerd://1.7.24
k0s-worker-1 Ready <none> 2m15s v1.31.3+k0s 10.81.146.51 <none> Alpine Linux v3.17 5.15.83-0-virt containerd://1.7.24
There is one node-local load balancer pod running for each worker node:
$ kubectl -n kube-system get pod -owide -l app.kubernetes.io/managed-by=k0s,app.kubernetes.io/component=nllb
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nllb-k0s-worker-0 1/1 Running 0 81s 10.81.146.198 k0s-worker-0 <none> <none>
nllb-k0s-worker-1 1/1 Running 0 85s 10.81.146.51 k0s-worker-1 <none> <none>
The cluster is using node-local load balancing and is able to tolerate the outage of one controller node. Shutdown the first controller to simulate a failure condition:
$ ssh -i k0s-ssh-private-key.pem k0s@10.81.146.254 'echo "Powering off $(hostname) ..." && sudo poweroff'
Powering off k0s-controller-0 ...
Node-local load balancing provides high availability from within the cluster,
not from the outside. The generated kubeconfig file lists the first controller's
IP as the Kubernetes API server address by default. As this controller is gone
by now, a subsequent call to kubectl will fail:
$ kubectl get nodes
Unable to connect to the server: dial tcp 10.81.146.254:6443: connect: no route to host
Changing the server address in k0s-kubeconfig from the first controller to
another one makes the cluster accessible again. Pick one of the other controller
IP addresses and put that into the kubeconfig file. The addresses are listed
both in k0sctl.yaml as well as in the output of kubectl -n default get
endpoints above.
$ ssh -i k0s-ssh-private-key.pem k0s@10.81.146.184 hostname
k0s-controller-1
$ sed -i s#https://10\\.81\\.146\\.254:6443#https://10.81.146.184:6443#g k0s-kubeconfig
$ kubectl get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k0s-worker-0 Ready <none> 3m35s v1.31.3+k0s 10.81.146.198 <none> Alpine Linux v3.17 5.15.83-0-virt containerd://1.7.24
k0s-worker-1 Ready <none> 3m34s v1.31.3+k0s 10.81.146.51 <none> Alpine Linux v3.17 5.15.83-0-virt containerd://1.7.24
$ kubectl -n kube-system get pods -owide -l app.kubernetes.io/managed-by=k0s,app.kubernetes.io/component=nllb
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nllb-k0s-worker-0 1/1 Running 0 2m31s 10.81.146.198 k0s-worker-0 <none> <none>
nllb-k0s-worker-1 1/1 Running 0 2m35s 10.81.146.51 k0s-worker-1 <none> <none>
The first controller is no longer active. Its IP address is not listed in the
default/kubernetes Endpoints resource and its k0s controller lease is orphaned:
$ kubectl -n default get endpoints
NAME ENDPOINTS AGE
kubernetes 10.81.146.113:6443,10.81.146.184:6443 3m56s
$ kubectl -n kube-node-lease get \
lease/k0s-ctrl-k0s-controller-0 \
lease/k0s-ctrl-k0s-controller-1 \
lease/k0s-ctrl-k0s-controller-2 \
lease/k0s-endpoint-reconciler
NAME HOLDER AGE
k0s-ctrl-k0s-controller-0 4m47s
k0s-ctrl-k0s-controller-1 fe45284924abb1bfce674e5a9aa8d647f17c81e53bbab17cf28288f13d5e8f97 4m28s
k0s-ctrl-k0s-controller-2 5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d 4m19s
k0s-endpoint-reconciler 5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d 4m47s
Despite that controller being unavailable, the cluster remains operational. The third controller has become the new k0s leader. Workloads will run just fine:
$ kubectl -n default run nginx --image=nginx
pod/nginx created
$ kubectl -n default get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 16s 10.244.0.5 k0s-worker-1 <none> <none>
$ kubectl -n default logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[notice] 1#1: using the "epoll" event method
[notice] 1#1: nginx/1.23.3
[notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
[notice] 1#1: OS: Linux 5.15.83-0-virt
[notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
[notice] 1#1: start worker processes
[notice] 1#1: start worker process 28