Durable Objects unbundled — what happens when a bucket becomes the durability layer, the coordination service, and the control plane all at once
• celld requires exactly four properties from a bucket — conditional create, conditional overwrite, read-after-write consistency, ranged reads — and several well-known 'S3-compatible' providers don't qualify.
• Ownership is a compare-and-swap loop: the bucket physically permits only one such write to succeed, so no separate lock service is needed.
• Fencing works two ways — a stale owner's writes land in an isolated, superseded epoch prefix, and each node independently self-fences when its own lease expires.
• celld relies on the object store's consistency guarantees. The provider operates the infrastructure behind those guarantees.
In the first post about celld, I described how object storage participates in coordination. I want to work through that mechanism here. How does a node claim a cell, and what happens if the previous owner is still running?
What celld needs from the object store
celld requires four properties from its object store:
- Conditional create — a write fails if the object already exists
- Conditional overwrite — a write fails if the object changed since it was last read
- Read-after-write consistency — a read immediately after a write returns that write, not a stale one
- Ranged reads — the store returns exactly the byte range requested
S3, Cloudflare R2, Google Cloud Storage, Azure Blob Storage, and Tigris meet all four. Backblaze B2, Hetzner, and DigitalOcean Spaces — despite being marketed as “S3-compatible” — do not.
For this design, API compatibility alone is not enough. We need to know what happens when two nodes try to update the same ownership record. The conditional-write semantics are part of the correctness of the system.
S3 added conditional writes in August 2024, followed by If-Match checks in November 2024. Those operations provide the compare-and-swap primitive used to arbitrate cell ownership.
Claiming ownership with compare-and-swap
Acquiring a cell reduces to one of two operations:
no record exists
│
▼
conditional create
│
▼
bucket accepts exactly onerecord exists, owner presumed gone
│
▼
compare-and-swap on the previous record
│
▼
bucket accepts exactly oneIf two nodes race to claim the same cell, both send a write, and the bucket’s own conditional-write semantics guarantee only one of them lands. A failed conditional write tells the competing node that its claim did not succeed.
The nodes use the bucket’s compare-and-swap operation to decide which claim succeeds. They do not run a separate consensus service for that decision.
Fencing with epochs and leases
If you know the classic distributed-locking literature — Kleppmann’s 2016 argument against Redlock — you’d expect fencing to work like this: the lock service hands out a monotonically increasing token, the client attaches it to every write, and the protected resource checks the token and rejects anything from a superseded holder.
celld uses the storage layout to isolate writes from different owners.
Every activation of a cell — a takeover, or a node waking a hibernated cell — bumps an epoch number. The epoch forms part of the storage path:
cells/agent-123/ltx/e94/ ← old owner's writes
cells/agent-123/ltx/e95/ ← current owner's writesSuppose a node loses ownership during a network partition but continues trying to write. Its writes still target e94/. The new owner writes to e95/. Recovery selects the current epoch’s lineage, so it needs to distinguish the history inherited from the previous owner from any stale writes made after takeover.
Kleppmann’s fencing requires the resource itself to be token-aware. celld’s version requires the reader to know which prefix is current — the isolation is structural, not enforced by a check on every write.
The design adds a second, complementary layer on top: self-fencing leases. Each node holds a lease in the bucket with a published expiry. If that expiry passes without renewal — because the node is partitioned, overloaded, or simply hasn’t heard back — the node doesn’t wait to be told it’s been superseded. It fences itself: stops the cell, fails every in-flight request it hasn’t completed.
lease expiry passes
│
▼
node stops the cell itself
│
▼
fails in-flight requests
│
▼
(does not wait to be told)A partitioned node may never receive a message telling it that another node has taken over. Lease expiry gives it a local condition for stopping work.
Who operates the coordination infrastructure?
The dependency on coordination remains. celld relies on the object store to provide the required guarantees.
S3’s read-after-write consistency guarantee — the one celld depends on for correctness — is itself the output of an internal, replicated metadata system that Amazon operates and you never see. Before December 2020, S3 didn’t even provide strong consistency by default; we cannot assume those semantics simply because a service stores objects.
Google’s Chubby lock service is a useful comparison: it exposes a lock API backed by a Paxos-based consensus system. The application uses the API while the service handles agreement between replicas. With celld, the object-store provider is responsible for the guarantees behind conditional writes.
your application
│
▼
celld
│
▼
object store API
│
▼
provider's internal
replicated metadata store
│
▼
provider manages
replication and agreementThe operational responsibility moves to the provider. We still need to understand its guarantees and decide how our application behaves when the service is unavailable.
What this trades away
I would look at three constraints before choosing this design:
Latency. Claiming a cell or renewing a lease requires a request to the object store. If the bucket is slow to respond, a new owner takes longer to start work. If a node cannot renew its lease before it expires, it has to stop serving its cells. So we avoid running a separate coordination service, but depend on the bucket responding promptly to keep cells available. This does not mean every application request waits for the bucket; these calls manage ownership and leases.
Opacity. You can’t debug the provider’s internal consensus system the way you can debug a self-hosted etcd cluster. When something goes wrong at that layer, you get an HTTP status code, not a Raft log to inspect.
Available operations. The primitive described here is a conditional update to one object. Requirements such as watching ownership changes or updating several ownership records atomically need additional mechanisms. This connects to the cross-cell transaction problem in the first post: deciding who owns each cell is only part of coordinating work across cells.
For me, the operational question is whether avoiding a separate consensus cluster is worth the dependency on the object store’s latency, availability and API. That depends on the workload and on what the team is already operating.
Do we need a separate coordination service?
The question I would take from this design is:
If your durability layer already offers a strong-enough write primitive, do you need a separate coordination service at all?
celld shows how conditional writes can support cell ownership using the same object store that holds durable state. The requirements are specific: we need to check the store’s guarantees against the operations the runtime actually performs.
I would start with the operations the application needs. If it only needs to claim ownership of individual cells, this is a design worth evaluating. If it needs watches or transactions across several ownership records, those requirements need to be addressed too. The smaller deployment does not remove that work.
The next post in this series takes apart the other half — how celld actually guarantees durability, and the LTX replication format underneath all of this.
Loading comments...