Published on

Deploying a Low-Overhead Kubernetes Distribution: K3S

Authors
  • avatar
    Name
    Timzaak
    Twitter

Rancher developed K3S, which is suitable for running a small internal Kubernetes cluster within a company. It has low maintenance costs and minimal resource overhead. However:

  1. K3S is designed with ultra-lightweight usage in mind, ideally running on a single machine. When used as a backend service orchestration layer, several aspects require adjustment.
  2. The docker.io registry is blocked in mainland China, preventing direct download of many associated container images.
  3. Rancher's latest versions come with K3S integrated, but they require access to docker.io. Due to point 2, switching to KuBoard is recommended as it is more friendly for deploying.

Key Considerations:

  1. Proxy Injection: If you set the http_proxy environment variable before installation, it will be injected into K3S, which can cause issues with pulling images. Be sure to remove it:
## Master
sudo vim /etc/systemd/system/k3s.service.env
sudo service k3s restart

## Worker
sudo vim /etc/systemd/system/k3s-agent.service.env
sudo service k3s-agent restart
  1. Remove Load from Master Node: K3S treats master nodes as worker nodes by default. To avoid scheduling workloads on the master:
kubectl taint nodes $node_name node-role.kubernetes.io/control-plane:NoSchedule
  1. Service LoadBalancer Node Selection: To avoid port conflicts (e.g., ports 80 and 443) on the master, specify which nodes the ServiceLB runs on:
### Reference: https://docs.k3s.io/networking/networking-services?_highlight=servicelb#controlling-servicelb-node-selection
kubectl label nodes ${node} svccontroller.k3s.cattle.io/enablelb=true
  1. CoreDNS Dependency: If CoreDNS fails, DNS resolution breaks entirely. Since images are pulled from docker.io, preload the necessary images on your machines.

  2. kubectl Config: On the master node, extract the kubectl config with:

sudo cat /etc/rancher/k3s/k3s.yaml

Remember to update the URL to match your actual server address.

  1. Importing Images: To preload container images:
sudo k3s ctr images import ./image.tar.gz

Download necessary images like rancher/mirrored, rancher/core-dns, or use air-gap packages from GitHub Releases. Be sure to match the version.

  1. Certificate Expiry: K3S certificates are valid for 1 year by default. Rotate them before expiry, or set up your own CA with a 10-year certificate:
# https://docs.k3s.io/cli/certificate
# Stop K3s
systemctl stop k3s

# Rotate certificates
k3s certificate rotate

# Start K3s
systemctl start k3s

Review After 6 Months:

  1. If using the cloud provider's built-in Load Balancer in production, there's no need to set up your own. The benefits of using Nginx or Higress may be negligible.
  2. Avoid using Docker Shim. Stick with the default container runtime. Using Docker will only introduce unnecessary shim images (docker ps clutter) and complicate maintenance.