K3s High Availability with Embedded etcd

A single K3s server on SQLite is one disk failure away from an unrecoverable control plane. High availability in K3s means three (or more) server nodes running embedded etcd — no external database, no load balancer requirement for agents, and one flag to start it.

The three topology choices

K3s supports three HA shapes, and the choice is made at cluster creation, not later:

Topology	Datastore	Who operates it	Best for
Single server	Embedded SQLite	K3s	Labs, edge sites, dev
3+ servers	Embedded etcd	K3s	Most HA needs
3+ servers	External MySQL/PostgreSQL	You	Existing DBA tooling, shared DB

The embedded option is the default recommendation because it keeps the whole system one artifact: etcd ships inside the k3s binary and is started and managed by the server process. You don’t install etcd, you don’t tune its flags separately, you don’t run it as a second cluster to babysit.

The external option exists for organizations that already run MySQL or PostgreSQL with backups, monitoring, and an on-call rotation. Point every server at the same DSN and K3s uses it instead. Note what that buys you: better DB operations, but now your Kubernetes control plane depends on a database cluster that K3s doesn’t manage.

Standing up the etcd cluster

The first server initializes the etcd cluster:

curl -sfL https://get.k3s.io | K3S_TOKEN=<secret> sh -s - \
  --cluster-init

--cluster-init tells the supervisor to bootstrap an embedded etcd cluster instead of creating a SQLite database. From there, additional servers join it:

# server two
curl -sfL https://get.k3s.io | K3S_URL=https://<server1>:6443 \
  K3S_TOKEN=<secret> sh -s - --server https://<server1>:6443

# equivalently, the explicit form:
k3s server \
  --server https://<server1>:6443 \
  --token <secret>

Every server must use the same token, and the joining server must reach :6443 on an existing member. Once joined, all three run a full control plane and share one etcd cluster underneath.

Two things that surprise people:

1. The token is cluster-wide. It authenticates both agents and server joins. Rotate it deliberately (--token on servers, --token on agents), not casually.
2. There is no “fixed registration address” for the datastore. The cluster’s identity is the first server’s URL you gave everyone else — which is why you should give servers a DNS name (k3s-1.internal) rather than an IP before you start. Agents and future servers both seed from it.

Quorum math: odd numbers only

Embedded etcd is a Raft cluster. Three members tolerate one failure; five tolerate two; four tolerates two failures in theory but can lose quorum with a single failure in some partitions — the classic mistake. Use 3 or 5, never 4.

• 3 servers — survives one node loss. Enough for most single-site HA.
• 5 servers — survives two, but doubles etcd write cost on every apiserver write (yes, every object write goes through Raft).
• More than 5 — etcd itself warns against it; write latency grows faster than availability.

Each server is a full control plane, so losing one also costs you an apiserver endpoint. That’s where agents come in.

Why agents don’t need a load balancer

In many Kubernetes HA setups you put a TCP load balancer in front of the apiservers so kubelets have one stable address. K3s doesn’t require that: each agent runs a client-side load balancer that tracks the cluster’s apiserver endpoints and fails over on connection failure. Combined with registration over a websocket seeded from the --server URL, an individual server outage is invisible to agents — they re-target a surviving member.

What you still might want a VIP or LB for:

• Your own kubectl from a laptop or CI — agents are self-sufficient, your tooling isn’t.
• External integrations (GitOps controllers, ingress controllers talking to the API from outside the cluster).

So: a load balancer is optional for cluster operation, recommended for everything else. Note it does not remove the need for the datastore to have quorum — if etcd loses quorum, all three apiservers stop committing writes regardless of how many are running.

Backups: k3s etcd-snapshot

Embedded etcd comes with snapshot tooling wired into the binary:

# take a manual snapshot
sudo k3s etcd-snapshot save --name pre-upgrade

# list snapshots
sudo k3s etcd-snapshot ls

# restore in place (server stopped)
sudo k3s etcd-snapshot restore --name pre-upgrade

Snapshots are also taken automatically on a schedule (configurable, default daily) and stored under /var/lib/rancher/k3s/server/db/snapshot/. Copy them off the node — a snapshot on the same disk as the cluster is not a backup.

The restore path is worth practicing once before you need it. And for external-datastore topologies, your backups are your database’s backups — K3s stops being responsible the moment you hand it a DSN.

What embedded etcd does and doesn’t give you

It gives you:

• Control-plane HA with zero extra software.
• Real Raft durability — snapshots, restore, membership.
• A single binary to upgrade, single version string to reason about.
• Tolerance for individual server failure, with agents failing over on their own.

It doesn’t give you:

• Datacenter HA. Three servers in one rack on one switch is one outage away from all three. Spread them across power/switch domains, or use an external database you already replicate.
• Separate etcd operations. You don’t get to run a dedicated 3-node etcd cluster with its own hardware, as upstream recommends for large clusters. Embedded etcd shares resources with the apiserver; on a heavily loaded control plane, that’s contention.
• Cross-cluster replication or multi-writer setups. One cluster, one datastore.
• Automatic SQLite → etcd migration. You choose at --cluster-init time, or you rebuild/migrate deliberately. Plan the topology before the first production workload lands.

The pragmatic read: embedded etcd takes K3s from “great edge toy” to “credible single-site HA control plane,” and the failure mode it eliminates — one server’s disk dying — is the one most small clusters actually hit. Beyond that, the remaining risks are physical, and no flag fixes a shared power strip.

Further reading

• K3s Architecture documentation
• K3s Datastore documentation