How celld Coordinates a Fleet Without a Separate Control Plane

The second post covered how a node claims a cell. The third covered how its writes survive a node failure. There is still a fleet to manage: how do the nodes find one another, share the cells and take over when a machine fails?

The details below refer to celld v0.5.1.

What a control plane does

A node is one running celld process. Nodes using the same bucket form a fleet. A cell is the named unit of application code and state that runs on one of those nodes.

The fleet needs to answer a few questions:

• Membership: which nodes are available?
• Failure handling: when can another node take over work?
• Placement: which node should own a cell, and when should it move?
• Discovery: where should a request for that cell go?

The software that makes these decisions is commonly called the control plane. In celld, the nodes perform this work using records in object storage. There is no separate control-plane service for the team to deploy.

The bucket is the directory

Suppose Node A starts. It has credentials to access the fleet’s bucket and a network address that other nodes can reach.

It writes a record containing its address and an expiry time. This is a lease: a registration that remains valid only while the node keeps renewing it. Other nodes read the record to find Node A and check whether its lease is still valid.

Node A starts
      │
      ▼
writes its address and expiry time
into a lease record in the bucket
      │
      ▼
Node B reads the record
      │
      ▼
Node B can contact Node A

Node leases tell us which machines are available. Separate cell ownership records tell us which node owns a particular cell, such as agent-123.

The lease also reports how many cells the node owns and how much memory and CPU it is using. These figures let the fleet share work and give operators information about when more capacity is needed.

Sharing cells between nodes

Each node has a target share of the fleet’s cells. That share is controlled by a placement weight, which defaults to the machine’s CPU count.

For example, with weights of 2 and 1, Node A’s target is roughly twice as many cells as Node B’s. The weight is a way to assign more cells to a larger machine. It does not mean every cell uses the same amount of CPU or memory.

By default, nodes read a shared summary of the fleet every five seconds. One node refreshes that summary from the individual lease records, and the others read the result. They use it to compare how many cells each node owns with its target.

If a node has more than its share, it can transfer cells to a node below its target. This balancing only moves hibernated cells: cells whose running code and database have been unloaded from memory. A cell still running in memory must hibernate before this balancing process can move it. The receiving node takes ownership but leaves the cell hibernated until work arrives. The balancing documentation describes the limits on each transfer.

celld does not start additional machines itself. The team, or an external scaling system, still decides when to add or remove nodes.

What happens when a node stops renewing its lease?

Suppose Node A owns agent-123, then loses its connection to the bucket. It may still be running and receiving requests, but it can no longer renew its lease.

Once the lease expires, Node A must stop serving its cells. Node B can then attempt to take ownership of agent-123.

Node A cannot renew its lease
        │
        ▼
    lease expires
        │
        ├── Node A stops serving its cells
        │
        ▼
Node B attempts to claim agent-123
        │
        ▼
bucket accepts a valid claim
        │
        ▼
Node B recovers the cell and starts it

If Node B and Node C both try to claim the same ownership record, the bucket’s conditional write accepts one of those competing updates. The new owner also gets a new epoch, a number identifying that activation of the cell. Its writes go under that epoch’s storage path, keeping them separate from writes attempted by an old owner. The second post explains that fencing mechanism.

Lease expiry does not tell us whether Node A crashed or merely lost its network connection. It gives the system a rule for when Node A must stop and another node may take over. celld implements that rule in the nodes and bucket records, without a separate failure-detection service. The ownership guarantees describe how a node stops work when its lease expires.

What bucket access allows

The fleet bucket stores several kinds of information:

fleet bucket
    │
    ├── application code and deployment records
    ├── cell data
    ├── cell ownership records
    ├── node lease records
    └── shared secret used to authenticate peers

A peer is another node in the fleet. Before accepting certain requests from a peer, a node needs to check that the sender has permission to act as a fleet node.

For that check, the nodes share a secret. The sender uses it to calculate an authentication code, called an HMAC, which the receiver verifies using the same secret. Someone who obtains that secret can produce valid codes too.

That is why the permissions on the bucket matter. Read access to cell data exposes that data. Read access to the shared secret can allow someone to impersonate a peer. Write access to deployment and ownership records can let someone change the code being run or interfere with who owns a cell.

The security documentation recommends giving each credential access to only one fleet bucket and replacing it after suspected disclosure. In practical terms, issue a replacement credential and revoke the exposed one.

The bucket credential and the shared peer secret are different things. Revoking bucket access does not make someone forget a secret they already copied. If that secret was exposed, replacing only the bucket credential would leave the peer-impersonation risk unresolved.

Authentication does not encrypt traffic

The HMAC check helps a node decide whether to accept a request. It does not hide the request’s contents from someone who can observe the network.

celld also opens connections between nodes to carry application requests. The request to open one of these tunnels is authenticated, but the calls sent through it use plain HTTP. The opening authentication check does not sign or encrypt the later traffic.

There is another reason to restrict network access: most internal operator routes, used to inspect and control a node, do not require request authentication. Peer-control routes and some runtime routes do use HMAC checks, but that protection does not cover the whole internal interface.

For v0.5.1, the documented deployment requirements are to keep the internal listener accessible only to trusted fleet nodes and operators, and to use an encrypted network such as WireGuard or Tailscale when the underlying network does not provide confidentiality. celld does not handle TLS itself. Public HTTPS must be provided by a proxy in front of it, with user authentication handled by the proxy or application.

What the team still operates

The team runs the celld nodes and manages their bucket access, network and deployment. It does not also need to run an etcd or ZooKeeper cluster for cell ownership and discovery.

There is still platform work around the runtime. The documented fleet model runs one application. It does not provide an account service for separate customers, a scheduler for several applications sharing the machines, or managed public routing and HTTPS. Those services must come from the surrounding deployment.

For me, the useful operational questions are concrete. How long do requests wait while a cell changes owner? Does a larger node receive the intended share of cells? Can an untrusted machine reach an operator endpoint? Can the team replace exposed credentials without leaving another route into the fleet?

A smaller set of services can make the deployment easier to operate. The test is whether the team can handle those failures and access changes reliably.