Run k0s in Docker#

You can create a k0s cluster on top of Docker.

Prerequisites#

You will require a Docker environment running on a Mac, Windows, or Linux system.

Container images#

The k0s OCI images are published to both Docker Hub and GitHub Container registry. For simplicity, the examples given here use Docker Hub (GitHub requires separate authentication, which is not covered here). The image names are as follows:

docker.io/k0sproject/k0s:v1.32.2-k0s.0
ghcr.io/k0sproject/k0s:v1.32.2-k0s.0

Note: Due to Docker's tag validation scheme, - is used as the k0s version separator instead of the usual +. For example, k0s version v1.32.2+k0s.0 is tagged as docker.io/k0sproject/k0s:v1.32.2-k0s.0.

Start k0s#

1. Run a controller#

By default, running the k0s OCI image will launch a controller with workloads enabled (i.e. a controller with the --enable-worker flag) to provide an easy local testing "cluster":

docker run -d --name k0s-controller --hostname k0s-controller \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --privileged `# this is the easiest way to enable container-in-container workloads` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:v1.32.2-k0s.0

Explanation of command line arguments:

-d runs the container in detached mode, i.e. in the background.
--name k0s-controller names the container "k0s-controller".
--hostname k0s-controller sets the hostname of the container to "k0s-controller".
-v /var/lib/k0s -v /var/log/pods creates two Docker volumes and mounts them to /var/lib/k0s and /var/log/pods respectively inside the container, ensuring that cluster data persists across container restarts.
--tmpfs /run TODO
--privileged gives the container the elevated privileges that k0s needs to function properly within Docker. See the section on adding additional workers for a more detailed discussion of privileges.
-p 6443:6443 exposes the container's Kubernetes API server port 6443 to the host, allowing you to interact with the cluster externally.
docker.io/k0sproject/k0s:v1.32.2-k0s.0 is the name of the k0s image to run.

By default, the k0s image starts a k0s controller with worker components enabled within the same container, creating a cluster with a single controller-and-worker node using the following command:

CMD ["k0s", "controller", "--enable-worker"]

Alternatively, a controller-only node can be run like this:

docker run -d --name k0s-controller --hostname k0s-controller \
  --read-only `# k0s won't write any data outside the below paths` \
  -v /var/lib/k0s `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --tmpfs /tmp `# allow writing temporary files` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:v1.32.2-k0s.0 \
  k0s controller

Note the addition of k0s controller to override the image's default command. Also note that a controller-only node requires fewer privileges.

2. (Optional) Add additional workers#

You can add multiple worker nodes to the cluster and then distribute your application containers to separate workers.

Acquire a join token for the worker:

token=$(docker exec k0s-controller k0s token create --role=worker)

Run the container to create and join the new worker:
```
docker run -d --name k0s-worker1 --hostname k0s-worker1 \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --privileged `# this is the easiest way to enable container-in-container workloads` \
  docker.io/k0sproject/k0s:v1.32.2-k0s.0 \
  k0s worker $token
```
Alternatively, with fine-grained privileges:
```
docker run -d --name k0s-worker1 --hostname k0s-worker1 \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --security-opt seccomp=unconfined \
  -v /dev/kmsg:/dev/kmsg:ro --device-cgroup-rule='c 1:11 r' \
  --cap-add sys_admin \
  --cap-add net_admin \
  --cap-add sys_ptrace \
  --cap-add sys_resource \
  docker.io/k0sproject/k0s:v1.32.2-k0s.0 \
  k0s worker "$token"
```
Notes on the security-related flags:
- --security-opt seccomp=unconfined is required for runc to access the session keyring.
- -v /dev/kmsg:/dev/kmsg:ro --device-cgroup-rule='c 1:11 r' allows reading of /dev/kmsg from inside the container. The kubelet's OOM watcher uses this.
Notes on Linux capabilities:
- CAP_SYS_ADMIN allows for a variety of administrative tasks, including mounting file systems and managing namespaces, which are necessary for creating and configuring nested containers.
- CAP_NET_ADMIN allows manipulation of network settings such as interfaces and routes, allowing containers to create isolated or bridged networks, and so on.
- CAP_SYS_PTRACE allows to inspect and modify processes, used to monitor other containers in a nested environment.
- CAP_SYS_RESOURCE allows containers to override resource limits for things like memory or file descriptors, used to manage and adjust resource allocation in nested container environments.
Note that more privileges may be required depending on your cluster configuration and workloads.

Repeat this step for each additional worker node and adjust the container and host names accordingly. Make sure that the workers can reach the controller on the required ports. If you are using Docker's default bridged network, this should be the case.

3. Access your cluster#

a) Using kubectl within the container#

To check cluster status and list nodes, use:

docker exec k0s-controller k0s kubectl get nodes

b) Using kubectl locally#

To configure local access to your k0s cluster, follow these steps:

Generate the kubeconfig:

docker exec k0s-controller k0s kubeconfig admin > ~/.kube/k0s.config

Update kubeconfig with Localhost Access:

To automatically replace the server IP with localhost dynamically in ~/.kube/k0s.config, use the following command:
```
sed -i '' -e "$(awk '/server:/ {print NR; exit}' ~/.kube/k0s.config)s|https://.*:6443|https://localhost:6443|" ~/.kube/k0s.config
```
This command updates the kubeconfig to point to localhost, allowing access to the API server from your host machine
Set the KUBECONFIG Environment Variable:
```
export KUBECONFIG=~/.kube/k0s.config
```
Verify Cluster Access:
```
kubectl get nodes
```

c) Use Lens #

Access the k0s cluster using Lens by following the instructions on how to add a cluster.

Use Docker Compose (alternative)#

As an alternative you can run k0s using Docker Compose:

services:
  k0s-controller:
    image: docker.io/k0sproject/k0s:v1.32.2-k0s.0
    container_name: k0s-controller
    hostname: k0s-controller
    network_mode: bridge # other modes are unsupported
    ports:
      - 6443:6443 # publish the Kubernetes API server port
    volumes:
      - /var/lib/k0s # this is where k0s stores its data
      - /var/log/pods # this is where k0s stores pod logs
      - /dev/kmsg:/dev/kmsg:ro # required by kubelets OOM watcher
    tmpfs:
      - /run # this is where k0s stores runtime data
    cap_add:
     - sys_admin
     - net_admin
     - sys_ptrace
     - sys_resource
    device_cgroup_rules:
      - c 1:11 r # required by kubelets OOM watcher
    configs:
      - source: k0s.yaml
        target: /etc/k0s/k0s.yaml

configs:
  k0s.yaml:
    content: |
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s
      spec:
        storage:
          type: kine
      # Any additional configuration goes here ...

Known limitations#

No custom Docker networks#

Currently, k0s nodes cannot be run if the containers are configured to use custom networks (for example, with --net my-net). This is because Docker sets up a custom DNS service within the network which creates issues with CoreDNS. No completely reliable workaounds are available, however no issues should arise from running k0s cluster(s) on a bridge network.

Next Steps#

Install using k0sctl: Deploy multi-node clusters using just one command
Control plane configuration options: Networking and datastore configuration
Worker node configuration options: Node labels and kubelet arguments
Support for cloud providers: Load balancer or storage configuration
Installing the Traefik Ingress Controller: Ingress deployment information