Upgrade etcd from 3.3 to 3.4

Processes, checklists, and notes on upgrading etcd from 3.3 to 3.4

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

NOTE: When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Highlighted breaking changes in 3.4.

Make `ETCDCTL_API=3 etcdctl` default

ETCDCTL_API=3 is now the default.

etcdctl set foo bar
Error: unknown command "set" for "etcdctl"

-etcdctl set foo bar
+ETCDCTL_API=2 etcdctl set foo bar
bar

ETCDCTL_API=3 etcdctl put foo bar
OK

-ETCDCTL_API=3 etcdctl put foo bar
+etcdctl put foo bar

Make `etcd --enable-v2=false` default

etcd --enable-v2=false is now the default.

This means, unless etcd --enable-v2=true is specified, etcd v3.4 server would not serve v2 API requests.

If v2 API were used, make sure to enable v2 API in v3.4:

-etcd
+etcd --enable-v2=true

Other HTTP APIs will still work (e.g. [CLIENT-URL]/metrics, [CLIENT-URL]/health, v3 gRPC gateway).

Deprecated `etcd --ca-file` and `etcd --peer-ca-file` flags

--ca-file and --peer-ca-file flags are deprecated; they have been deprecated since v2.1.

Note setting this parameter will also automatically enable client cert authentication no matter what value is set for --client-cert-auth.

-etcd --ca-file ca-client.crt
+etcd --trusted-ca-file ca-client.crt

-etcd --peer-ca-file ca-peer.crt
+etcd --peer-trusted-ca-file ca-peer.crt

Deprecated `grpc.ErrClientConnClosing` error

grpc.ErrClientConnClosing has been deprecated in gRPC >= 1.10.

import (
+	"go.etcd.io/etcd/clientv3"

	"google.golang.org/grpc"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
)

_, err := kvc.Get(ctx, "a")
-if err == grpc.ErrClientConnClosing {
+if clientv3.IsConnCanceled(err) {

// or
+s, ok := status.FromError(err)
+if ok {
+  if s.Code() == codes.Canceled

Require `grpc.WithBlock` for client dial

The new client balancer uses an asynchronous resolver to pass endpoints to the gRPC dial function. As a result, v3.4 client requires grpc.WithBlock dial option to wait until the underlying connection is up.

import (
	"time"
	"go.etcd.io/etcd/clientv3"
+	"google.golang.org/grpc"
)

+// "grpc.WithBlock()" to block until the underlying connection is up
ccfg := clientv3.Config{
  Endpoints:            []string{"localhost:2379"},
  DialTimeout:          time.Second,
+ DialOptions:          []grpc.DialOption{grpc.WithBlock()},
  DialKeepAliveTime:    time.Second,
  DialKeepAliveTimeout: 500 * time.Millisecond,
}

Deprecating `etcd_debugging_mvcc_db_total_size_in_bytes` Prometheus metrics

v3.4 promotes etcd_debugging_mvcc_db_total_size_in_bytes Prometheus metrics to etcd_mvcc_db_total_size_in_bytes, in order to encourage etcd storage monitoring.

etcd_debugging_mvcc_db_total_size_in_bytes is still served in v3.4 for backward compatibilities. It will be completely deprecated in v3.5.

-etcd_debugging_mvcc_db_total_size_in_bytes
+etcd_mvcc_db_total_size_in_bytes