How celld Makes SQLite Writes Durable

The first post in this series introduced the write path: local SQLite, LTX replication, then an acknowledgement once the durability condition is met. The second post covered ownership. Here I want to separate those two responsibilities and work through what has to survive for an acknowledged write to be recovered.

The implementation details below were checked against celld v0.5.1, released on 19 September 2026.

Where LTX comes from

LTX, for Lite Transaction file, was built by Fly.io for LiteFS, its FUSE-based system for replicating SQLite across a cluster.

SQLite stores a database in fixed-size blocks called pages. Updating a row changes one or more of those pages. A transaction groups changes together so they either all take effect or none of them do, even if the process crashes halfway through.

To make that possible, SQLite keeps a separate record of data needed to recover from an interrupted transaction. This is what the journal is for.

A rollback journal stores the original contents of pages before SQLite overwrites them in the database file. If the transaction fails, SQLite can copy those originals back and undo the changes. For example, if a transaction changes a balance from £100 to £80, the journal preserves the page containing the previous balance so SQLite can restore it if necessary.

In SQLite’s DELETE journal mode, SQLite saves the database changes to disk and then deletes the journal to mark the transaction as committed. Other journal modes mark completion differently.

LiteFS sits between SQLite and the filesystem, so it can observe these writes and the request to delete the journal. At that point, it collects the updated pages into an LTX file. The rollback journal holds the old data needed to undo a transaction; the LTX file holds the new data needed to apply it to another copy of the database.

before overwriting database pages
        │
        ▼
save original pages in rollback journal
        │
        ▼
write updated pages to database
        │
        ▼
SQLite requests journal deletion to commit
        │
        ▼
LiteFS captures updated pages in LTX
        │
        ▼
     journal deleted

The LTX file includes checksums to detect corruption and a transaction ID that increases with each transaction, so another node can track which changes it has applied.

SQLite also supports a write-ahead log, or WAL. In that mode, it writes new page contents to a separate log before copying them into the main database file later. In WAL mode, SQLite marks the final log entry to show that the transaction is complete. LiteFS uses that marker to identify the changes to package into an LTX file.

Litestream — a separate, older project for streaming SQLite backups to object storage — adopted the same format in its 0.5 rewrite, replacing its previous approach of shipping raw WAL segments.

Compaction reduces restore work

Before version 0.5, Litestream copied chunks of SQLite’s write-ahead log to object storage. Restoring a database meant loading the most recent snapshot, then replaying the log changes saved since that snapshot. Moving to LTX made it possible to combine those changes before a restore needed them.

celld uses this approach too. Its replication library originated from a Rust reimplementation of Litestream, and its compactor builds larger LTX files from smaller ones.

Suppose the same database page changes twenty times. To reconstruct its state at the end of those changes, we only need the final version. Compaction puts that version into a combined file, along with the latest versions of the other pages changed in the same range of transactions.

smaller LTX files
(each contains changed pages)
        │
        ▼
merge the files, keeping the newest
version of each page in that range
        │
        ▼
one larger LTX file
        │
        ▼
fewer files and page versions
needed during restore

In celld v0.5.1, this runs in the background and is enabled by default. A cell becomes eligible for compaction when its durable transaction ID has advanced by 256 since the last compaction, or when 32 MiB of changes have accumulated. These thresholds are configurable. The runtime combines the initial files, called L0, into larger L1 files. The scheduler uses those thresholds, rather than Litestream’s timed sequence of 30-second, five-minute and hourly levels.

The compaction operation retains its source files. It adds a combined file that a restore can use in their place.

How celld rebuilds the database

The restore planner selects a snapshot where one is available, then adds files covering the changes after it. It chooses files that extend the transaction history without leaving gaps. A combined LTX file can cover a range that would otherwise require many smaller files; newer changes that have not yet been compacted are included too.

For a smaller restore, celld downloads the selected files and uses their page contents to reconstruct a local SQLite database.

For a larger restore, it can start without downloading all the database contents first. It builds an index showing where each page can be found, fetches a page from object storage when SQLite first needs it, and fills the rest of the local database in the background. This is called paged restore. The default threshold is 256 MiB of selected restore files, provided the fleet supports the format. The restore implementation chooses between these two paths.

Compaction reduces how much history a restore has to read. Paged restore reduces how much database content must be downloaded before the cell can start serving requests.

When celld acknowledges a write

celld always waits for evidence that a write has been saved before reporting success. The CELLD_DURABILITY setting selects which copies it can rely on.

Fleet mode — the default, which can use other nodes as well as object storage:

write arrives at owning node
        │
        ├─────────────────────────┐
        ▼                         ▼
send to other nodes       upload to object storage
        │                         │
        ▼                         ▼
required copies          object store confirms
saved to disk            the upload
        │                         │
        └────────────┬────────────┘
                     ▼
         whichever finishes first
         allows a success response

Usually, saving the change on other nodes lets the caller receive a response before the bucket upload finishes. Until that upload completes, recovery of the change depends on those copies surviving.

Bucket mode — explicitly selected with CELLD_DURABILITY=bucket:

write arrives
        │
        ▼
sent to object storage
        │
        ▼
object store confirms the upload
        │
        ▼
caller receives success

Bucket mode always waits for object storage, even when other nodes are available. In fleet mode, a node with no replication peers also waits for object storage because it has nowhere else to save a copy.

In fleet mode, a successful write can still be lost if every node holding a copy loses its data before the upload finishes. Losing just one node should not lose the write.

In bucket mode, celld reports success only after object storage has saved the change. Losing the celld node after that does not lose the write; recovery depends on the stored copy remaining available.

The useful distinction is between ownership and durability. The bucket arbitrates who owns the cell. Peer replication provides a surviving copy of what that owner writes. Those are separate responsibilities, even though both are needed to recover the cell after a failure.

The settings that control the wait

CELLD_DURABILITY defaults to fleet. Setting it to bucket makes every write wait for object storage, as described above.

CELLD_LTX_DURABILITY_TIMEOUT_SECS, default 10, sets the time allowed to obtain the required confirmation once upload work starts. A write can also spend time queued behind other cells, so this is not a ten-second limit on the whole HTTP request. The configuration documentation describes those separate waits.

For an application, I would check how long a caller might wait, what response it receives if that wait fails, and whether retrying could repeat work already committed locally.

Earlier versions exposed CELLD_OUTPUT_GATE as a way to disable the durability wait. That setting has been removed. In v0.5.1, celld always waits for the configured durability confirmation before acknowledging a write.

Recovery is part of normal operation

If the last snapshot was taken at 10:00 and the node loses its data at 10:05, restoring that snapshot will not recover the changes made during those five minutes. Those changes need to have been saved elsewhere. Restore time also depends on the amount of data to download and any additional history available to replay.

In celld, replicated state is used when a new owner resumes a cell. Before restoring from the bucket, the new owner checks whether the previous owner could have acknowledged writes that had not yet been uploaded.

If so, the takeover recovery process collects the retained changes from reachable replication peers and uploads them to the bucket before allowing the restore to proceed. This ensures the restore includes those acknowledged changes rather than starting from an older bucket copy.

That makes replication part of normal cell movement and recovery, as well as protection against machine failure.

What I would test with an application

The mechanisms are described in the code, but I would still test them with the application’s workload. How long does a busy cell take to recover? How do large writes affect the response time? What happens to requests during a rolling restart, when fewer replication peers are available?

For paged restore, I would also measure the first few requests after takeover. The cell can start before every page is local, so those requests may need to fetch data from object storage.

Ownership and durability are separate responsibilities

The previous post described how conditional writes and epochs establish cell ownership. This post adds the write path: the owner waits for the required copies on other nodes, or a completed bucket upload, before telling the caller the write succeeded.

For me, the practical question is which copies exist when a response reaches the caller, and which failures those copies survive. That is what I would test before relying on the RPO target.

The final post looks at how nodes discover each other and distribute work, and what access to the fleet bucket allows.