Configuration options#

Using a configuration file#

k0s can be installed without a config file. In that case the default configuration will be used. You can, though, create and run your own non-default configuration (used by the k0s controller nodes).

k0s supports providing only partial configurations. In case of partial configuration is provided, k0s will use the defaults for any missing values.

Generate a yaml config file that uses the default settings.

mkdir -p /etc/k0s
k0s config create > /etc/k0s/k0s.yaml

Modify the new yaml config file according to your needs, refer to Configuration file reference below. You can remove the default values if wanted as k0s supports partial configs too.

Install k0s with your new config file.

sudo k0s install controller -c /etc/k0s/k0s.yaml

If you need to modify your existing configuration later on, you can change your config file also when k0s is running, but remember to restart k0s to apply your configuration changes.
```
sudo k0s stop
sudo k0s start
```

Configuring k0s via k0sctl#

k0sctl can deploy your configuration options at cluster creation time. Your options should be placed in the spec.k0s.config section of the k0sctl's configuration file. See the section on how to install k0s via k0sctl and the k0sctl README for more information.

Configuration file reference#

CAUTION: As many of the available options affect items deep in the stack, you should fully understand the correlation between the configuration file components and your specific environment before making any changes.

A YAML config file follows, with defaults as generated by the k0s config create command:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  api:
    address: 192.168.68.104
    externalAddress: my-lb-address.example.com
    k0sApiPort: 9443
    port: 6443
    sans:
    - 192.168.68.104
  controllerManager: {}
  extensions:
    helm:
      concurrencyLevel: 5
      charts: null
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    calico: null
    clusterDomain: cluster.local
    dualStack: {}
    kubeProxy:
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      ipMasq: false
      metricsPort: 8080
      mtu: 0
      peerRouterASNs: ""
      peerRouterIPs: ""
      extraArgs:
    nodeLocalLoadBalancing:
      enabled: false
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.244.0.0/16
    provider: kuberouter
    serviceCIDR: 10.96.0.0/12
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 192.168.68.104
    type: etcd
  telemetry:
    enabled: true
  featureGates:
    - name: feature_XXX
      enabled: true
      components: ["kubelet", "kube-api", "kube-scheduler"]
    - name: feature_YYY
      enabled: true
    -
      name: feature_ZZZ
      enabled: false

`spec` Key Detail#

`spec.api`#

Element	Description
`externalAddress`	The loadbalancer address (for k0s controllers running behind a loadbalancer). Configures all cluster components to connect to this address and also configures this address for use when joining new nodes to the cluster.
`address`	Local address on which to bind an API. Also serves as one of the addresses pushed on the k0s create service certificate on the API. Defaults to first non-local address found on the node.
`sans`	List of additional addresses to push to API servers serving the certificate.
`extraArgs`	Map of key-values (strings) for any extra arguments to pass down to Kubernetes api-server process.
`port`¹	Custom port for kube-api server to listen on (default: 6443)
`k0sApiPort`¹	Custom port for k0s-api server to listen on (default: 9443)

¹ If port and k0sApiPort are used with the externalAddress element, the loadbalancer serving at externalAddress must listen on the same ports.

`spec.storage`#

Element	Description
`type`	Type of the data store (valid values:`etcd` or `kine`). Note: Type `etcd` will cause k0s to create and manage an elastic etcd cluster within the controller nodes.
`etcd.peerAddress`	Node address used for etcd cluster peering.
`etcd.extraArgs`	Map of key-values (strings) for any extra arguments to pass down to etcd process.
`kine.dataSource`	kine datasource URL.
`etcd.externalCluster`	Configuration when etcd is externally managed, i.e. running on dedicated nodes. See `spec.storage.etcd.externalCluster`

`spec.storage.etcd.externalCluster`#

k0s can also work with externally managed Etcd cluster. If this is configured, k0s will NOT set up etcd, it has to be managed manually.

Element	Description
`endpoints`	Array of Etcd endpoints to use.
`etcdPrefix`	Prefix to use for this cluster. The same external Etcd cluster can be used for several k0s clusters, each prefixed with unique prefix to store data with.
`caFile`	CaFile is the host path to a file with Etcd cluster CA certificate.
`clientCertFile`	ClientCertFile is the host path to a file with TLS certificate for etcd client.
`clientKeyFile`	ClientKeyFile is the host path to a file with TLS key for etcd client.

`spec.network`#

Element	Description
`provider`	Network provider (valid values: `calico`, `kuberouter`, or `custom`). For `custom`, you can push any network provider (default: `kuberouter`). Be aware that it is your responsibility to configure all of the CNI-related setups, including the CNI provider itself and all necessary host levels setups (for example, CNI binaries). Note: Once you initialize the cluster with a network provider the only way to change providers is through a full cluster redeployment.
`podCIDR`	Pod network CIDR to use in the cluster.
`serviceCIDR`	Network CIDR to use for cluster VIP services.
`clusterDomain`	Cluster Domain to be passed to the kubelet and the coredns configuration.

`spec.network.calico`#

Element	Description
`mode`	`vxlan` (default), `ipip` or `bird`
`overlay`	Overlay mode: `Always` (default), `CrossSubnet` or `Never` (requires `mode=vxlan` to disable calico overlay-network).
`vxlanPort`	The UDP port for VXLAN (default: `4789`).
`vxlanVNI`	The virtual network ID for VXLAN (default: `4096`).
`mtu`	MTU for overlay network (default: `0`, which causes Calico to detect optimal MTU during bootstrap).
`wireguard`	Enable wireguard-based encryption (default: `false`). Your host system must be wireguard ready (refer to the Calico documentation for details).
`flexVolumeDriverPath`	The host path for Calicos flex-volume-driver(default: `/usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds`). Change this path only if the default path is unwriteable (refer to Project Calico Issue #2712 for details). Ideally, you will pair this option with a custom `volumePluginDir` in the profile you use for your worker nodes.
`ipAutodetectionMethod`	Use to force Calico to pick up the interface for pod network inter-node routing (default: `""`, meaning not set, so that Calico will instead use its defaults). For more information, refer to the Calico documentation.
`envVars`	Map of key-values (strings) for any calico-node environment variable.

`spec.network.calico.envVars`#

Environment variable's value must be string, e.g.:

spec:
  network:
    provider: calico
    calico:
      envVars:
        TEST_BOOL_VAR: "true"
        TEST_INT_VAR: "42"
        TEST_STRING_VAR: test

K0s runs Calico with some predefined vars, which can be overwritten by setting new value in spec.network.calico.envVars:

CALICO_IPV4POOL_CIDR: "{{ spec.network.podCIDR }}"
CALICO_DISABLE_FILE_LOGGING: "true"
FELIX_DEFAULTENDPOINTTOHOSTACTION: "ACCEPT"
FELIX_LOGSEVERITYSCREEN: "info"
FELIX_HEALTHENABLED: "true"
FELIX_PROMETHEUSMETRICSENABLED: "true"
FELIX_FEATUREDETECTOVERRIDE: "ChecksumOffloadBroken=true"

FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true disables VXLAN offloading because of projectcalico/calico#4727.

In SingleStack mode there are additional vars:

FELIX_IPV6SUPPORT: "false"

In DualStack mode there are additional vars:

CALICO_IPV6POOL_NAT_OUTGOING: "true"
FELIX_IPV6SUPPORT: "true"
IP6: "autodetect"
CALICO_IPV6POOL_CIDR: "{{ spec.network.dualStack.IPv6podCIDR }}"

`spec.network.kuberouter`#

Element	Description
`autoMTU`	Autodetection of used MTU (default: `true`).
`mtu`	Override MTU setting, if `autoMTU` must be set to `false`).
`metricsPort`	Kube-router metrics server port. Set to 0 to disable metrics (default: `8080`).
`peerRouterIPs`	DEPRECATED: Use extraArgs with peerRouterIPs instead. Comma-separated list of global peer addresses.
`peerRouterASNs`	DEPRECATED: Use extraArgs with peerRouterASNs instead. Comma-separated list of global peer ASNs.
`hairpin`	Hairpin mode, supported modes `Enabled`: enabled cluster wide, `Allowed`: must be allowed per service using annotations, `Disabled`: doesn't work at all (default: Enabled)
`hairpinMode`	Deprecated Use `hairpin` instead. If both `hairpin` and `hairpinMode` are defined, this is ignored. If only hairpinMode is configured explicitly activates hairpinMode (https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode).
`ipMasq`	IP masquerade for traffic originating from the pod network, and destined outside of it (default: false)
`extraArgs`	Extra arguments to pass to kube-router. Can be also used to override any k0s managed args. For reference, see kube-router documentation. (default: empty)

Note: Kube-router allows many networking aspects to be configured per node, service, and pod (for more information, refer to the Kube-router user guide).

`spec.network.kubeProxy`#

Element	Description
`disabled`	Disable kube-proxy altogether (default: `false`).
`mode`	Kube proxy operating mode, supported modes `iptables`, `ipvs`, `userspace` (default: `iptables`)
`iptables`	Kube proxy iptables settings
`ipvs`	Kube proxy ipvs settings
`nodePortAddresses`	Kube proxy nodePortAddresses

Default kube-proxy iptables settings:

iptables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s

Default kube-proxy ipvs settings:

ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s

`spec.network.nodeLocalLoadBalancing`#

Configuration options related to k0s's node-local load balancing feature.

Element	Description
`enabled`	Indicates if node-local load balancing should be used to access Kubernetes API servers from worker nodes. Default: `false`.
`type`	The type of the node-local load balancer to deploy on worker nodes. Default: `EnvoyProxy`. (This is the only option for now.)
`envoyProxy`	Configuration options related to the "EnvoyProxy" type of load balancing.

`spec.network.nodeLocalLoadBalancing.envoyProxy`#

Configuration options required for using Envoy as the backing implementation for node-local load balancing.

Note: This type of load balancing is not supported on ARMv7 workers.

Element	Description
`image`	The OCI image that's being used for the Envoy Pod.
`imagePullPolicy`	The pull policy being used used for the Envoy Pod. Defaults to `spec.images.default_pull_policy` if omitted.
`apiServerBindPort`	Port number on which to bind the Envoy load balancer for the Kubernetes API server to on a worker's loopback interface. Default: `7443`.
`konnectivityServerBindPort`	Port number on which to bind the Envoy load balancer for the konnectivity server to on a worker's loopback interface. Default: `7132`.

`spec.network.controlPlaneLoadBalancing`#

Configuration options related to k0s's control plane load balancing feature

Element	Description
`enabled`	Indicates if control plane load balancing should be enabled. Default: `false`.
`type`	The type of the control plane load balancer to deploy on controller nodes. Currently, the only supported type is `Keepalived`.
`keepalived`	Contains the keepalived configuration.

`spec.network.controlPlaneLoadBalancing.Keepalived`#

Configuration options related to keepalived in control plane load balancing

Element	Description
`vrrpInstances`	Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs.
`virtualServers`	Configuration options related LoadBalancing. This is an array which allows to configure multiple LBs.

`spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`#

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

Element	Description
`virtualIPs`	VirtualIPs is the list of virtual IP address used by the VRRP instance. Each virtual IP must be a CIDR as defined in RFC 4632 and RFC 4291.A list of the CIDRs handled by the VRRP instance.
`interface`	The NIC used by the virtual router. If not specified, k0s will use the interface that owns the default route.
`virtualRouterID`	The VRRP router ID. If not specified, k0s will automatically number the IDs for each VRRP instance, starting with 51. It must be in the range of 1-255, all the control plane nodes must use the same `virtualRouterID`. Other clusters in the same network must not use the same `virtualRouterID`.
`advertIntervalSeconds`	Advertisement interval in seconds. Defaults to 1 second.
`authPass`	The password for accessing VRRPD. This is not a security feature but a way to prevent accidental misconfigurations. It must be in the range of 1-8 characters

`spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`#

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

Element	Description
`ipAddress`	The load balancer's listen address.
`delayLoop`	Delay timer for check polling. DelayLoop accepts microsecond precision. Further precision will be truncated without warnings. Defaults to `1m`.
`lbAlgo`	Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`.
`lbKind`	Kind of ipvs load balancer. Supported values: `NAT`, `DR`, `TUN` Default: `DR`.
`persistenceTimeoutSeconds`	Timeout value for persistent connections in seconds. Must be in the range of 1-2678400 (31 days). If not specified, defaults to 360 (6 minutes).

`spec.controllerManager`#

Element	Description
`extraArgs`	Map of key-values (strings) for any extra arguments you want to pass down to the Kubernetes controller manager process.

`spec.scheduler`#

Element	Description
`extraArgs`	Map of key-values (strings) for any extra arguments you want to pass down to Kubernetes scheduler process.

`spec.workerProfiles`#

Worker profiles are used to manage worker-specific configuration in a centralized manner. A ConfigMap is generated for each worker profile. Based on the --profile argument given to the k0s worker, the configuration in the corresponding ConfigMap is is picked up during startup.

The worker profiles are defined as an array. Each element has following properties:

Property	Description
`name`	String; name to use as profile selector for the worker process
`values`	Object; Kubelet configuration overrides, see below for details

`spec.workerProfiles[].values` (Kubelet configuration overrides)#

The Kubelet configuration overrides of a profile override the defaults defined by k0s.

Note that there are several fields that cannot be overridden:

clusterDNS
clusterDomain
apiVersion
kind
staticPodURL

Configuration examples#

Custom volumePluginDir#

spec:
  workerProfiles:
    - name: custom-pluginDir
      values:
         volumePluginDir: /var/libexec/k0s/kubelet-plugins/volume/exec

Eviction Policy#

spec:
  workerProfiles:
    - name: custom-eviction
      values:
        evictionHard:
          memory.available: "500Mi"
          nodefs.available: "1Gi"
          imagefs.available: "100Gi"
        evictionMinimumReclaim:
          memory.available: "0Mi"
          nodefs.available: "500Mi"
          imagefs.available: "2Gi"

Unsafe Sysctls#

spec:
  workerProfiles:
    - name: custom-eviction
      values:
        allowedUnsafeSysctls:
          - fs.inotify.max_user_instances

`spec.featureGates`#

Available components are:

kube-apiserver
kube-controller-manager
kubelet
kube-scheduler
kube-proxy

If components are omitted, propagates to all kube components.

Modifies extraArgs.

Examples#

Generic feature gates example#

spec:
    featureGates:
      - name: feature-gate-0
        enabled: true
        components: ["kube-apiserver", "kube-controller-manager", "kubelet", "kube-scheduler"]
      - name: feature-gate-1
        enabled: true
      - name: feature-gate-2
        enabled: false

Kubelet feature gates example#

The below is an example of a k0s config with feature gates enabled:

spec:
    featureGates:
      - name: DevicePlugins
        enabled: true
        components: ["kubelet"]
      - name: Accelerators
        enabled: true
        components: ["kubelet"]
      - name: AllowExtTrafficLocalEndpoints
        enabled: false

`spec.images`#

Nodes under the images key all have the same basic structure:

spec:
  images:
    coredns:
      image: quay.io/coredns/coredns
      version: v1.7.0

If you want the list of default images and their versions to be included, use k0s config create --include-images.

Available keys#

spec.images.konnectivity
spec.images.metricsserver
spec.images.kubeproxy
spec.images.coredns
spec.images.pause
spec.images.calico.cni
spec.images.calico.flexvolume
spec.images.calico.node
spec.images.calico.kubecontrollers
spec.images.kuberouter.cni
spec.images.kuberouter.cniInstaller
spec.images.repository¹

¹ If spec.images.repository is set and not empty, every image will be pulled from images.repository

If spec.images.default_pull_policy is set and not empty, it will be used as a pull policy for each bundled image.

Image example#

images:
  repository: "my.own.repo"
  konnectivity:
    image: calico/kube-controllers
    version: v3.16.2
  metricsserver:
    image: registry.k8s.io/metrics-server/metrics-server
    version: v0.7.2

In the runtime the image names are calculated as my.own.repo/calico/kube-controllers:v3.16.2 and my.own.repo/metrics-server/metrics-server:v0.7.2. This only affects the the imgages pull location, and thus omitting an image specification here will not disable component deployment.

`spec.extensions.helm`#

spec.extensions.helm is the config file key in which you configure the list of Helm repositories and charts to deploy during cluster bootstrap (for more information, refer to Helm Charts).

`spec.extensions.storage`#

spec.extensions.storage controls bundled storage provider. The default value external makes no storage deployed.

To enable embedded host-local storage provider use the following configuration:

spec:
  extensions:
    storage:
      type: openebs_local_storage

`spec.konnectivity`#

The spec.konnectivity key is the config file key in which you configure Konnectivity-related settings.

agentPort agent port to listen on (default 8132)
adminPort admin port to listen on (default 8133)

`spec.telemetry`#

To improve the end-user experience k0s is configured by defaul to collect telemetry data from clusters and send it to the k0s development team. To disable the telemetry function, change the enabled setting to false.

The telemetry interval is ten minutes.

spec:
  telemetry:
    enabled: true

Disabling controller components#

k0s allows to completely disable some of the system components. This allows users to build a minimal Kubernetes control plane and use what ever components they need to fulfill their need for the control plane. Disabling the system components happens through a command line flag for the controller process:

--disable-components strings                     disable components (valid items: applier-manager,autopilot,control-api,coredns,csr-approver,endpoint-reconciler,helm,konnectivity-server,kube-controller-manager,kube-proxy,kube-scheduler,metrics-server,network-provider,node-role,system-rbac,windows-node,worker-config)

Note: As of k0s 1.26, the kubelet-config component has been replaced by the worker-config component. k0s will issue a warning when the old component name is being used. It is scheduled for removal in k0s 1.27. Please update to the new component name.

If you use k0sctl, just add the flag when installing the cluster for the first controller at spec.hosts.installFlags in the config file like e.g.:

spec:
  hosts:
  - role: controller
    installFlags:
    - --disable-components=metrics-server

As seen from the component list, the only always-on component is the Kubernetes api-server, without that k0s serves no purpose.

Configuration options#

Using a configuration file#

Configuring k0s via k0sctl#

Configuration file reference#

spec Key Detail#

spec.api#

spec.storage#

spec.storage.etcd.externalCluster#

spec.network#

spec.network.calico#

spec.network.calico.envVars#

spec.network.kuberouter#

spec.network.kubeProxy#

spec.network.nodeLocalLoadBalancing#

spec.network.nodeLocalLoadBalancing.envoyProxy#

spec.network.controlPlaneLoadBalancing#

spec.network.controlPlaneLoadBalancing.Keepalived#

spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances#

spec.network.controlPlaneLoadBalancing.keepalived.virtualServers#

spec.controllerManager#

spec.scheduler#

spec.workerProfiles#

spec.workerProfiles[].values (Kubelet configuration overrides)#