The server ends a Lock stream at grant (`sender_stream` is taken), so the
hold task always saw "disconnected" right after acquiring and re-acquired
with the original request_id. Each re-bind got a stream that ended at once,
and the Bound path had no backoff: every held lock spun at hundreds of Lock
RPCs per second (~350/s measured direct, ~10k in 3s from one leptin pod at
boot). On leptin prod (2026-10-04) the flood starved ExtendLease, leases
lapsed by TTL (430 expiries on one node in 5h), and the spinning re-acquire
re-won them under fresh ids: the holder's generation changed under it, a
second pod could be granted the key meanwhile, and leptin-server — which
treats a generation change as loss — crash-looped for 13h.
- After the stream ends, the hold task never re-acquires. It re-proves
ownership via lease_status every lease/3 (200ms..5s) and publishes `lost`
when it cannot.
- The server reports a missing lease as Unavailable (indistinguishable from
a transport error), so `Lost` was unreachable and a lapsed lease was never
reported. An unproven lease past its last proven expiry is now `lost`.
- Removes the now-unused re-acquire / orphan-release path.
Verified live against the leptin prod cluster through the istio mesh: two
pods contending on one key for 3 min — 36 polls, 0 generation changes,
mutual exclusion held; a 3s lease without renewal is reported lost at 3.0s.
The Go and TS clients still use the re-bind design and are NOT fixed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MCJdDXt4eEwJc7HHiUrM8d
A released lock could come back as a ghost. `Lock::unlock` signalled the
background hold task and sent UnLock immediately; if the task had an
idempotent re-acquire RPC in flight, `race_stop` dropped the future
client-side but the server kept processing it — once UnLock had removed
the original lease, the same request_id re-WON the key with a fresh id
that no client held. On graylin prod that ghost writer sat on
`shutdown:pod:<id>:barrier` (created ~1 ms after "released :barrier
writer", then "lease_ttl pq has a removed node" and never reaped) and
parked every proxied request on the pod for the full 120 s restart wait.
- hold_loop no longer races the acquire RPC against stop; it lets the
RPC finish (bounded by HOLD_RPC_TIMEOUT), then honours a level
`stopped` flag and releases any lease it won while stopping.
- unlock() sets `stopped`, notifies, JOINS the hold task (abort only as
a backstop past every bound), then UnLocks the *current* id.
- Drop sets the flag and notifies but no longer aborts mid-RPC.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MCwe5Sf5NU8Ei13oqTcesj
Extract the waymaker client into a standalone, polyglot repo versioned in
lockstep with the waymaker server (this tag == server v0.1.27).
- proto/ vendored client-facing protos (source of truth: waymaker repo).
internal_proxy (server-internal ProxyService + replication) is
intentionally excluded — it is a server concern, not a client one.
- rust/ standalone crate; build.rs generates from proto/. Quorum fence
scope + Lock auto-reacquire / fence-watch (watch/is_lost) API.
- go/ module git.awesomike.com/pub/waymaker-client/go — generated stubs
+ ergonomic wrappers (locks/streams/kv/collections/sketches/object).
- ts/ @waymaker/client — generated stubs + ergonomic wrappers.
- scripts/ sync-protos.sh (re-vendor + align VERSION) + gen-go.sh + gen-ts.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Michael Netshipise