Blog on etcd

Avoiding Zombie Cluster Members When Upgrading to etcd v3.6

Wed, 17 Dec 2025 00:00:00 +0000

Summary: SIG-etcd has patched another potential issue blocking upgrades from v3.5 to v3.6. If you are upgrading, make sure to update to v3.5.26 or later first.

Issue Summary

Recently, the etcd community addressed an issue that may appear when users upgrade from v3.5 to v3.6. This bug can cause the cluster to report “zombie members”, which are etcd nodes that were removed from the database cluster some time ago, and are re-appearing and joining database consensus. The etcd cluster is then inoperable until these zombie members are removed.

Follow Up - Preventing Upgrade Failures from etcd v3.5 to v3.6

Tue, 21 Oct 2025 00:00:00 +0000

We have identified and fixed an additional scenario that may cause upgrade failures when moving from etcd v3.5 to v3.6. This post contains details, the fix, and additional workarounds. Please refer to issue 20793 to get detailed technical information.

Issue

In a previous post — How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6 — we described an upgrade issue affecting etcd versions in v3.5.1-v3.5.19. That issue was addressed in v3.5.20. However, a follow-up investigation revealed that the original fix did not cover all scenarios.

Autonomous Testing of etcd's Robustness

Fri, 03 Oct 2025 00:00:00 +0000

This is a post from the CNCF blog which we are sharing with our community as well.

As a critical component of many production systems, including Kubernetes, the etcd project’s first priority is reliability. Ensuring consistency and data safety requires our project contributors to continuously improve testing methodologies. In this article, we will describe how we used advanced simulation testing to uncover subtle bugs, validate the robustness of our releases, and increase our confidence in etcd’s stability. We’ll share our key findings and how they have improved etcd.

Announcing etcd v3.6.0

Thu, 15 May 2025 00:00:00 +0000

Introduction
Security
Features
Performance
- Memory
- Throughput
Breaking changes
Testing
Critical bug fixes
Upgrade issue
Platforms
Dependencies
Community
Extended Support for v3.4
Future Development

Introduction

Today, we are releasing etcd v3.6.0, the first minor release since etcd v3.5.0 on June 15, 2021. This release introduces several new features, makes significant progress on long-standing efforts like downgrade support and migration to v3store, and addresses numerous critical & major issues. It also includes major optimizations in memory usage, improving efficiency and performance.

How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6

Thu, 27 Mar 2025 00:00:00 +0000

Update (October 21, 2025)

We have identified and fixed two additional upgrade issues. Please see our new blog posts Follow Up - Preventing Upgrade Failures from etcd v3.5 to v3.6 and Avoiding Zombie Cluster Members When Upgrading for details.

There is a common issue 19557 in the etcd v3.5 to v3.6 upgrade that may cause the upgrade process to fail. You can find detailed information and related discussions in the issue.

TL; DR

Users are required to first upgrade to etcd v3.5.20 (or a higher patch version) before upgrading to etcd v3.6.0. Failure to do so may result in an unsuccessful upgrade.

Engage with the etcd project at KubeCon NA 2023

Sun, 05 Nov 2023 00:00:00 +0000

KubeCon NA 2023 in Chicago is just around the corner! This year, the etcd project has a diverse range of talks, tutorials, and even interactive contribfest sessions for you to get involved in . As a critical foundational pillar of the Kubernetes ecosystem, etcd’s presence at Kubecon underscores its importance in ensuring all our Kubernetes clusters continue to have robust and reliable distributed persistent state.

Here’s a detailed overview of what you can expect from the Etcd Project’s presence at KubeCon NA 2023:

Introducing sig-etcd

Wed, 01 Nov 2023 00:00:00 +0000

Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, new SIGs can be created, and that was precisely what happened recently.

SIG etcd is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans.

The critical role of etcd

If we look inside the control plane of a Kubernetes cluster, we will find etcd, a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data -- this description alone highlights the critical role that etcd plays, and the importance of it within the Kubernetes ecosystem.

How to debug large db size issue?

Wed, 04 Jan 2023 00:00:00 +0000

Background

Users can configure the quota of the backend db size using flag --quota-backend-bytes. It’s the max number of bytes the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the suggested max value is 8GB.

2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message etcdserver: mvcc: database space exceeded when trying to write more data, and see alarm “NOSPACE” (see example below) when checking the endpoint status or health state. It would be better to figure out whether it’s expected. It’s exactly the reason why I provide this guide.

etcd Integrates Continuous Fuzzing

Fri, 11 Mar 2022 00:00:00 +0000

In the last few months, the team at Ada Logics has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s value for etcd both short term and long term. All fuzzers were implemented by way of go-fuzz and when running in OSS-Fuzz instrumented by way of libFuzzer, and as such, etcd uses state-of-the-art open source fuzzing capabilities. The full report of the engagement can be found here.

Announcing etcd 3.5

Tue, 15 Jun 2021 00:00:00 +0000

When we launched etcd 3.4 back in August 2019, our focus was on storage backend improvements, non-voting member and pre-vote features. Since then, etcd has become more widely used for various mission critical clustering and database applications and as a result, its feature set has grown more broad and complex. Thus, improving its stability and reliability has been top priority in recent development.

Today, we are releasing etcd 3.5. The past two years allowed for extensive iterations in fixing numerous bugs, identifying optimization opportunities at scale, and evolving its surrounding ecosystem. The etcd project also became a CNCF graduated project during this time frame. This release is the result of continuous evolution and grungy, thankless tasks done by the etcd community.

Latest Jepsen Results against etcd 3.4.3

Thu, 30 Jan 2020 00:00:00 +0000

Jepsen tested and analyzed etcd 3.4.3, and had both good results and useful feedback to share with us.

A key part of etcd’s design is strong consistency guarantees across the distributed key-value store. Kubernetes, Rook, OpenStack, and countless other critical software projects rely on etcd, in part, because of the etcd project’s focus on reliability and correctness.

Over the years, the etcd team has put tremendous effort on building testing and chaos engineering frameworks. We feel that we’ve improved our consistency, and have asked Jepsen for validation.

Container Linux with systemd

Tue, 23 May 2017 00:00:00 +0000

This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services. The original page was authored by Gyuho Lee.

The following blog post shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

Cluster bootstrapping in Container Linux is simplest with Ignition; coreos-metadata.service dynamically fetches the machine’s IP for discovery. Note that etcd’s discovery service protocol is only meant for bootstrapping, and cannot be used with runtime reconfiguration or cluster monitoring.

Deployment of etcd on AWS EC2

Mon, 10 Apr 2017 00:00:00 +0000

This post provides an introduction to design considerations when designing an etcd deployment on AWS EC2 and how AWS specific features may be utilized in that context. Also, this post assumes operational knowledge of Amazon Web Services (AWS), specifically Amazon Elastic Compute Cloud (EC2).

FreeBSD

Thu, 23 Jan 2014 00:00:00 +0000

Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now etcd and etcdctl can be enjoyed on FreeBSD 10.0 (RC4 as of now) and 9.x, where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but it is untested; caveat emptor.