45e5f89c77
The server ends a Lock stream at grant (`sender_stream` is taken), so the hold task always saw "disconnected" right after acquiring and re-acquired with the original request_id. Each re-bind got a stream that ended at once, and the Bound path had no backoff: every held lock spun at hundreds of Lock RPCs per second (~350/s measured direct, ~10k in 3s from one leptin pod at boot). On leptin prod (2026-10-04) the flood starved ExtendLease, leases lapsed by TTL (430 expiries on one node in 5h), and the spinning re-acquire re-won them under fresh ids: the holder's generation changed under it, a second pod could be granted the key meanwhile, and leptin-server — which treats a generation change as loss — crash-looped for 13h. - After the stream ends, the hold task never re-acquires. It re-proves ownership via lease_status every lease/3 (200ms..5s) and publishes `lost` when it cannot. - The server reports a missing lease as Unavailable (indistinguishable from a transport error), so `Lost` was unreachable and a lapsed lease was never reported. An unproven lease past its last proven expiry is now `lost`. - Removes the now-unused re-acquire / orphan-release path. Verified live against the leptin prod cluster through the istio mesh: two pods contending on one key for 3 min — 36 polls, 0 generation changes, mutual exclusion held; a 3s lease without renewal is reported lost at 3.0s. The Go and TS clients still use the re-bind design and are NOT fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCJdDXt4eEwJc7HHiUrM8d |
||
|---|---|---|
| go | ||
| proto | ||
| rust | ||
| scripts | ||
| ts | ||
| .gitignore | ||
| Cargo.toml | ||
| README.md | ||
| VERSION | ||
README.md
waymaker-client
Official client libraries for waymaker — the distributed coordination service (locks, streams, KV, collections, sketches, cache, object store) over gRPC.
Three clients, one wire contract, versioned in lockstep with the waymaker server:
| Language | Path | Package |
|---|---|---|
| Rust | rust/ |
waymaker-client (crate) |
| Go | go/ |
git.awesomike.com/pub/waymaker-client/go (module) |
| TypeScript | ts/ |
@waymaker/client (npm) |
The .proto files in proto/ are vendored copies; the waymaker server
repo is the source of truth. scripts/sync-protos.sh refreshes them and keeps
VERSION aligned with the server.
Versioning
Every release is tagged at the same version as the waymaker server it
targets (see VERSION). A client tagged v0.1.27 speaks the wire contract of
waymaker v0.1.27. The gRPC wire format is backward-compatible across patch
releases (enum integer values are stable), so a client one patch behind a
server generally interoperates — but match versions for new surface.
Layout
proto/ vendored .proto (source of truth: waymaker repo)
rust/ Rust client crate (hand-written ergonomic wrappers + generated stubs)
go/ Go client module
ts/ TypeScript client (npm)
scripts/ sync-protos.sh + codegen scripts (gen-go/gen-ts)
VERSION lockstep version with the waymaker server
Regenerating
# refresh protos from a local waymaker checkout, then regenerate all stubs
WAYMAKER_REPO=../waymaker ./scripts/sync-protos.sh
./scripts/gen-go.sh
./scripts/gen-ts.sh
# Rust regenerates from proto/ automatically via build.rs on `cargo build`.
Quickstart
- Rust — see
rust/README.md. Leader election with the fence-watch / auto-reacquireLock. - Go — see
go/README.md. - TypeScript — see
ts/README.md.