TSE Queries Should Degrade Instead of Dying

Introduction

Query paths are useful precisely when something is broken.

The problem is that a two-tier store doubles the ways a query can fail. If the planner picks exactly one tier and that tier is the broken one, or if any tier error fails the whole request, then a cold-file corruption or a hot-table lock takes down reads that most of the data could have answered.

I wanted a different default for TSE's router:

A dead tier costs freshness. It never costs the whole answer.

New to TSE? Start with What TSE Is: SQLite Hot, Parquet Cold — this post assumes the vocabulary from that overview but recaps what it needs.

That led to a deliberately boring design: fan out to both tiers around the watermark, overlap the boundary by 10 minutes, merge with dedup in one pass, and return tier faults as Warnings[] — never a 500.

No query planner. No distributed transaction. No fail-closed errors.

The goal isn't literally free availability. The goal is to make partial truth cheap enough to always return that failing the whole query is harder to justify than answering from what is reachable.

The tax nobody notices

A two-tier query sounds like pure upside until an error fails it closed.

A typical naive version looks roughly like this:

The query now depends on its weakest tier. If the cold reader hits a corrupt file, the hot rows sitting right there never reach the caller. If the watermark just advanced mid-query, rows near the boundary belong to both tiers and get counted twice — or neither.

So teams often make the router conditional:

"We'll retry when the tier recovers."

That is backwards. The incident is exactly when you need the rows that are still readable. I wanted the router to answer from whatever is reachable, say what it couldn't reach, and let the caller decide — because the caller knows whether stale is acceptable, and the router doesn't.

One rule: degrade, don't die

The architecture is built around one invariant:

Tier faults arrive as warnings. Empty answers arrive only when both tiers are actually empty.

The router reads the watermark, splits the time range with a ±10min overlap around it, queries both tiers concurrently, clamps each side to the overlap edge, and merges:

The database can be corrupt without the query dying. That is the whole idea.

The shape of the code

The implementation is intentionally small:

boundary := wm.LastTS
overlap := int64(10 * time.Minute) // microseconds
needHot := q.SinceUs == 0 || q.UntilUs == 0 || q.UntilUs > boundary-overlap
needCold := q.UntilUs == 0 || q.SinceUs < boundary+overlap
// Fan out with WaitGroup, clamp each tier to the overlap edge,
// cold error -> Warnings, then:
result.Events = kWayMergeDedup(hotEvents, coldEvents, q.Limit)

There are three decisions here.

1. Overlap, not a clean cut

The watermark moves while queries run. A clean hot/cold split would risk double-counting or dropping rows near the boundary. The 10-minute overlap means boundary rows are read from both tiers and deduplicated by ID downstream:

Duplication is handled once, in the merge — not scattered across callers.

2. Cold failure is a warning

A missing or invalid Parquet file appends "cold tier: …" to Warnings[] and the query continues with hot rows. Hot errors warn the same way. The contract in tse/types.go makes this explicit: Result carries Events + Warnings + Cursor + Total, and AddWarning exists precisely so partial results are first-class, not exceptional.

3. Merge once, with the limit pushed down

Both tiers return ID-ordered rows (UUIDv7 sorts as time), so the merge is a single heap pass with dedup and an early stop at Limit — capped by ApplyDefaults at 100k against OOM. The cold reader additionally stops scanning files past 2 * Limit, and streams row groups in 4k batches instead of materializing them. The cursor returned is just the last ID seen, so QueryWithCursor continues where the page ended.

Why these boring tools?

A sync.WaitGroup instead of a pipeline framework: two goroutines, one merge, no scheduler to debug. A 5-second watermark cache (invalidated on flush/compact via InvalidateCache) instead of a subscription: staleness is bounded and explicit. A semaphore-bounded ReaderPool (default 4, swappable via SetReader if DuckDB ever returns) instead of unbounded parallelism: cold I/O degrades gracefully under fan-out rather than stampeding the disk.

Every piece fails in the same direction: slower or staler, never a dead query.

What the benchmark says

From results-2026-09-23.csv, 50k-event runs on a single Windows/NVMe box, pure-Go:

The shape follows fan-out math: with hot p50 $p_h$ and cold p50 $p_c$, merged p50 $\approx \max(p_h, p_c)$ — the overlap window bounds how much both tiers scan, so the slower tier sets the pace and the faster one rides along.

That is a regression guard, not a universal claim: slower disks and heavier overlap change the constants. What doesn't change is the policy — the numbers measure speed, while Warnings[] measures survival.

Failure should move in one direction

If the cold tier is corrupt, the query answers from hot and says so. If hot is locked, cold answers and says so. Three drills I actually ran, all green:

• Corrupt a Parquet file → scrubber marks it, queries warn and continue from hot
• SIGKILL mid-flush → restart replays from the watermark, merged result == 50000, zero duplicates
• Fill the disk → writes reject with ErrDiskFull, committed rows still query clean

The system degrades from:

fresh and complete

to:

stale but committed, with warnings

rather than from:

available

to:

The honest limit

The design has real weaknesses. The overlap double-scans ~20 minutes of boundary data on every cross-tier query — cheap at 10 minutes each way, but not zero. The 5-second watermark cache means a query can miss rows flushed seconds ago; if you need read-your-write, invalidate the cache. The 100k DefaultMaxLimit cap means huge exports paginate by cursor or they don't happen. And Warnings[] puts the burden on the caller: a dashboard that ignores warnings will silently show partial data.

That means the router chooses:

never fail a query a subset of tiers can answer

over:

never show a partial answer.

For an event store queried during incidents, that is the current invariant. But it is still an invariant with a cost: callers must read the warnings.

The actual goal

The goal was never to make queries literally infallible.

Disks fail. Files corrupt. Watermarks move mid-read.

The goal was to make the cost of a broken tier lower than the cost of a broken query — cheap enough to always answer from what is reachable that failing everything is never the default.

That requires one architectural rule:

Partial failure is data, not an exception. Put it in the result, not in the error path.

For TSE, fan-out, a 10-minute overlap, one merge pass, and a warnings array are enough to make that practical. No planner required. No 500 required.

And most importantly, the query does not have to wait for its slowest tier to be healthy before answering. That's the kind of read path I want over a two-tier store: boring, concurrent, measurable, and difficult to accidentally take down.