- Published on
Deploying a Low-Overhead Kubernetes Distribution: K3S
- Authors
- Name
- Timzaak
Rancher developed K3S, which is suitable for running a small internal Kubernetes cluster within a company. It has low maintenance costs and minimal resource overhead. However:
- K3S is designed with ultra-lightweight usage in mind, ideally running on a single machine. When used as a backend service orchestration layer, several aspects require adjustment.
- The
docker.io
registry is blocked in mainland China, preventing direct download of many associated container images. - Rancher's latest versions come with K3S integrated, but they require access to
docker.io
. Due to point 2, switching to KuBoard is recommended as it is more friendly for deploying.
Key Considerations:
- Proxy Injection: If you set the
http_proxy
environment variable before installation, it will be injected into K3S, which can cause issues with pulling images. Be sure to remove it:
## Master
sudo vim /etc/systemd/system/k3s.service.env
sudo service k3s restart
## Worker
sudo vim /etc/systemd/system/k3s-agent.service.env
sudo service k3s-agent restart
- Remove Load from Master Node: K3S treats master nodes as worker nodes by default. To avoid scheduling workloads on the master:
kubectl taint nodes $node_name node-role.kubernetes.io/control-plane:NoSchedule
- Service LoadBalancer Node Selection: To avoid port conflicts (e.g., ports 80 and 443) on the master, specify which nodes the ServiceLB runs on:
### Reference: https://docs.k3s.io/networking/networking-services?_highlight=servicelb#controlling-servicelb-node-selection
kubectl label nodes ${node} svccontroller.k3s.cattle.io/enablelb=true
CoreDNS Dependency: If CoreDNS fails, DNS resolution breaks entirely. Since images are pulled from
docker.io
, preload the necessary images on your machines.kubectl Config: On the master node, extract the kubectl config with:
sudo cat /etc/rancher/k3s/k3s.yaml
Remember to update the URL to match your actual server address.
- Importing Images: To preload container images:
sudo k3s ctr images import ./image.tar.gz
Download necessary images like rancher/mirrored
, rancher/core-dns
, or use air-gap packages from GitHub Releases. Be sure to match the version.
- Certificate Expiry: K3S certificates are valid for 1 year by default. Rotate them before expiry, or set up your own CA with a 10-year certificate:
# https://docs.k3s.io/cli/certificate
# Stop K3s
systemctl stop k3s
# Rotate certificates
k3s certificate rotate
# Start K3s
systemctl start k3s
Review After 6 Months:
- If using the cloud provider's built-in Load Balancer in production, there's no need to set up your own. The benefits of using Nginx or Higress may be negligible.
- Avoid using Docker Shim. Stick with the default container runtime. Using Docker will only introduce unnecessary shim images (
docker ps
clutter) and complicate maintenance.