Version v3.6-DRAFT of the documentation is in DRAFT status. For the latest stable documentation, see v3.5.
Downgrade etcd from v3.6 to v3.5
In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:
- one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
- after enabling the downgrade, new features in v3.6 are no longer available to the cluster
Before starting a downgrade, read through the rest of this guide to prepare.
Downgrade checklists
Highlighted breaking changes from v3.6 to v3.5:
Difference in flags
If you are using any of the following flags in your v3.6 configurations, make sure to remove, rename, or change the default value when downgrading to v3.5.
NOTE: The diff is based on version v3.6.0 and v.3.5.18. The actual diff would be dependent on your patch version, check with diff <(etcd-3.6/bin/etcd -h | grep \\-\\-) <(etcd-3.5/bin/etcd -h | grep \\-\\-) first.
# flags not available in v3.5
-etcd --discovery-token ''
-etcd --discovery-endpoints ''
-etcd --discovery-dial-timeout '2s'
-etcd --discovery-request-timeout '5s'
-etcd --discovery-keepalive-time '2s'
-etcd --discovery-keepalive-timeout '6s'
-etcd --discovery-insecure-transport 'true'
-etcd --discovery-insecure-skip-tls-verify 'false'
-etcd --discovery-cert ''
-etcd --discovery-key ''
-etcd --discovery-cacert ''
-etcd --discovery-user ''
-etcd --discovery-password ''
-etcd --feature-gates
-etcd --log-format
# same flag with different names
-etcd --bootstrap-defrag-threshold-megabytes
+etcd --experimental-bootstrap-defrag-threshold-megabytes
-etcd --compaction-batch-limit
+etcd --experimental-compaction-batch-limit
-etcd --compact-hash-check-time
+etcd --experimental-compact-hash-check-time
-etcd --compaction-sleep-interval
+etcd --experimental-compaction-sleep-interval
-etcd --corrupt-check-time
+etcd --experimental-corrupt-check-time
-etcd --enable-distributed-tracing
+etcd --experimental-enable-distributed-tracing
-etcd --distributed-tracing-address
+etcd --experimental-distributed-tracing-address
-etcd --distributed-tracing-instance-id
+etcd --experimental-distributed-tracing-instance-id
-etcd --distributed-tracing-sampling-rate
+etcd --experimental-distributed-tracing-sampling-rate
-etcd --distributed-tracing-service-name
+etcd --experimental-distributed-tracing-service-name
-etcd --downgrade-check-time
+etcd --experimental-downgrade-check-time
-etcd --max-learners
+etcd --experimental-max-learners
-etcd --memory-mlock
+etcd --experimental-memory-mlock
-etcd --peer-skip-client-san-verification
+etcd --experimental-peer-skip-client-san-verification
-etcd --snapshot-catchup-entries
+etcd --experimental-snapshot-catchup-entries
-etcd --warning-apply-duration
+etcd --experimental-warning-apply-duration
-etcd --warning-unary-request-duration
+etcd --experimental-warning-unary-request-duration
-etcd --watch-progress-notify-interval
+etcd --experimental-watch-progress-notify-interval
# equivalent flags of v3.6 feature gates
-etcd --feature-gates=CompactHashCheck=true
+etcd --experimental-compact-hash-check-enabled=true
-etcd --feature-gates=InitialCorruptCheck=true
+etcd --experimental-enable-initial-corrupt-check=true
-etcd --feature-gates=LeaseCheckpoint=true
+etcd --experimental-enable-lease-checkpoint=true
-etcd --feature-gates=LeaseCheckpointPersist=true
+etcd --experimental-enable-lease-checkpoint-persist=true
-etcd --feature-gates=StopGRPCServiceOnDefrag=true
+etcd --experimental-stop-grpc-service-on-defrag=true
-etcd --feature-gates=TxnModeWriteWithSharedBuffer=false
+etcd --experimental-txn-mode-write-with-shared-buffer=false
# same flag different defaults
-etcd --snapshot-count=10000
+etcd --snapshot-count=100000
-etcd --v2-deprecation='write-only'
+etcd --v2-deprecation='not-yet'
-etcd --discovery-fallback='exit'
+etcd --discovery-fallback='proxy'
Difference in Prometheus metrics
# metrics not available in v3.5
-etcd_network_known_peers
-etcd_server_feature_enabled
Server downgrade checklists
Downgrade requirements
To ensure a smooth rolling downgrade, the running cluster must be healthy. Check the health of the cluster by using the etcdctl endpoint health command before proceeding.
Preparation
Before downgrading etcd, always test the services relying on etcd in a staging environment before deploying the downgrade to the production environment.
Before beginning, download the snapshot backup. Should something go wrong with the downgrade, it is possible to use this backup to rollback back to existing etcd version.
Before beginning, download the latest release of etcd v3.5.
Mixed versions
While downgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is considered downgraded once downgrade is enabled by etcdctl downgrade enable 3.5. Internally, the overall cluster version is set to the downgrade target version, which controls the reported version and the supported features.
Rollback
Before downgrading your etcd cluster, please create and download a snapshot backup of your etcd cluster. This snapshot can be used to restore the cluster to its pre-upgrade state if needed. If users encounter issues during the downgrade, they should first identify and resolve the root cause.
If the downgrade has started after running etcdctl downgrade enabled, and the cluster is still in a mixed-version state—where at least one member remains on v3.6, users can cancel the ongoing downgrade process by running etcdctl downgrade cancel, and restarting all the downgraded members with the original v3.6 binaries.
Once all members have been downgraded to v3.5, the cluster is considered fully downgraded. If users wish to return to the original version after a full downgrade has completed, they should follow the official upgrade guide to ensure consistency and avoid data corruption.
Downgrade procedure
This example shows how to downgrade a 3-member v3.6 etcd cluster running on a local machine.
Step 1: check downgrade requirements
Is the cluster healthy and running v3.6.x?
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint health
<<COMMENT
localhost:2379 is healthy: successfully committed proposal: took = 2.118638ms
localhost:22379 is healthy: successfully committed proposal: took = 3.631388ms
localhost:32379 is healthy: successfully committed proposal: took = 2.157051ms
COMMENT
curl http://localhost:2379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
curl http://localhost:22379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
curl http://localhost:32379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 10 | 10 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 10 | 10 | | | false |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 10 | 10 | | | false |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
Step 2: download snapshot backup from leader
Download the snapshot backup to provide a downgrade path should any problems occur.
Step 3: validate downgrade target version
Validate the downgrade target version before enabling the downgrade:
- We only support downgrading one minor version at a time. e.g downgrading from v3.6 to v3.4 isn’t allowed.
- Please do not move on to next step until the validation is successful.
etcdctl downgrade validate 3.5
<<COMMENT
Downgrade validate success, cluster version 3.6
COMMENT
Step 4: enable downgrade
etcdctl downgrade enable 3.5
<<COMMENT
Downgrade enable success, cluster version 3.6
COMMENT
After enabling downgrade, the cluster will start to operate with v3.5 protocol, which is the downgrade target version. In addition, etcd will automatically migrate the schema to the downgrade target version, which usually happens very fast. Confirm the storage version of all servers has been migrated to v3.5 by checking the endpoint status before moving on to the next step.
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
NOTE: Once downgrade is enabled, the cluster will remain operating with v3.5 protocol even if all the servers are still running the v3.6 binary, unless the downgrade is canceled with etcdctl downgrade cancel
Step 5: stop one existing etcd server
Before stopping the server, check if it is the leader. We recommend downgrading the leader last.
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
If the server to be stopped is the leader, you can avoid some downtime by move-leader to another server before stopping this server.
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 move-leader 91bc3c398fb3c146
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 13 | 13 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 13 | 13 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 13 | 13 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:
{"level":"warn","ts":"2025-02-28T17:35:43.795069Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://127.0.0.1:12380/version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:43.795149Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:44.368651Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8211f1d0f64f3269","rtt":"483.01µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:44.368726Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"8211f1d0f64f3269","rtt":"735.659µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}
Step 6: restart the etcd server with same configuration (minus the flags that are removed or replaced in v3.5)
Restart the etcd server with same configuration but with the new etcd binary.
-etcd-3.6/bin --name s1 \
+etcd-3.5/bin --name s1 \
--data-dir /tmp/etcd/s1 \
--listen-client-urls http://localhost:2379 \
--advertise-client-urls http://localhost:2379 \
--listen-peer-urls http://localhost:2380 \
--initial-advertise-peer-urls http://localhost:2380 \
--initial-cluster s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380 \
--initial-cluster-token tkn \
--initial-cluster-state existing
Verify that each member, and then the entire cluster, becomes healthy with the new v3.5 etcd binary:
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 14 | 14 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 14 | 14 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 14 | 14 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
<<COMMENT
localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
COMMENT
NOTE: You will see the DOWNGRADE ENABLED is false for the v3.5 server, because the downgrade info is not implemented in v3.5 status endpoint, downgrade is still enabled for the cluster at this point.
Step 7: repeat step 5 and step 6 for rest of the members
When all members are downgraded, check the health and status of the cluster, and confirm the minor version of all members is v3.5, and storage version is empty:
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 26 | 26 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 26 | 26 | | | false |
| localhost:32379 | fd422379fda50e48 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 26 | 26 | | | false |
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
<<COMMENT
localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
COMMENT
curl http://localhost:2379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
curl http://localhost:22379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
curl http://localhost:32379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
In the log of the leader, you should be able to see message similar to the following:
{"level":"info","ts":"2025-02-28T17:59:50.019862Z","caller":"etcdserver/server.go:2749","msg":"the cluster has been downgraded","cluster-version":"3.5.0"}
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.