0 Releases 6 Tags

• v0.1.33 45e5f89c77

  Compare

  fix(lock): a held lock polls ownership instead of re-acquiring in a spin (v0.1.33)

  0 commits to main since this tag

  The server ends a Lock stream at grant (sender_stream is taken), so the
  hold task always saw "disconnected" right after acquiring and re-acquired
  with the original request_id. Each re-bind got a stream that ended at once,
  and the Bound path had no backoff: every held lock spun at hundreds of Lock
  RPCs per second (~350/s measured direct, ~10k in 3s from one leptin pod at
  boot). On leptin prod (2026-10-04) the flood starved ExtendLease, leases
  lapsed by TTL (430 expiries on one node in 5h), and the spinning re-acquire
  re-won them under fresh ids: the holder's generation changed under it, a
  second pod could be granted the key meanwhile, and leptin-server — which
  treats a generation change as loss — crash-looped for 13h.

  • After the stream ends, the hold task never re-acquires. It re-proves
    ownership via lease_status every lease/3 (200ms..5s) and publishes lost
    when it cannot.
  • The server reports a missing lease as Unavailable (indistinguishable from
    a transport error), so Lost was unreachable and a lapsed lease was never
    reported. An unproven lease past its last proven expiry is now lost.
  • Removes the now-unused re-acquire / orphan-release path.

  Verified live against the leptin prod cluster through the istio mesh: two
  pods contending on one key for 3 min — 36 polls, 0 generation changes,
  mutual exclusion held; a 3s lease without renewal is reported lost at 3.0s.

  The Go and TS clients still use the re-bind design and are NOT fixed here.

  Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
  Claude-Session: https://claude.ai/code/session_01MCJdDXt4eEwJc7HHiUrM8d

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)