waymaker-client/rust
Michael Netshipise 45e5f89c77 fix(lock): a held lock polls ownership instead of re-acquiring in a spin (v0.1.33)
The server ends a Lock stream at grant (`sender_stream` is taken), so the
hold task always saw "disconnected" right after acquiring and re-acquired
with the original request_id. Each re-bind got a stream that ended at once,
and the Bound path had no backoff: every held lock spun at hundreds of Lock
RPCs per second (~350/s measured direct, ~10k in 3s from one leptin pod at
boot). On leptin prod (2026-10-04) the flood starved ExtendLease, leases
lapsed by TTL (430 expiries on one node in 5h), and the spinning re-acquire
re-won them under fresh ids: the holder's generation changed under it, a
second pod could be granted the key meanwhile, and leptin-server — which
treats a generation change as loss — crash-looped for 13h.

- After the stream ends, the hold task never re-acquires. It re-proves
  ownership via lease_status every lease/3 (200ms..5s) and publishes `lost`
  when it cannot.
- The server reports a missing lease as Unavailable (indistinguishable from
  a transport error), so `Lost` was unreachable and a lapsed lease was never
  reported. An unproven lease past its last proven expiry is now `lost`.
- Removes the now-unused re-acquire / orphan-release path.

Verified live against the leptin prod cluster through the istio mesh: two
pods contending on one key for 3 min — 36 polls, 0 generation changes,
mutual exclusion held; a 3s lease without renewal is reported lost at 3.0s.

The Go and TS clients still use the re-bind design and are NOT fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MCJdDXt4eEwJc7HHiUrM8d
2026-10-04 14:54:45 +02:00
..
src fix(lock): a held lock polls ownership instead of re-acquiring in a spin (v0.1.33) 2026-10-04 14:54:45 +02:00
Cargo.toml fix(lock): a held lock polls ownership instead of re-acquiring in a spin (v0.1.33) 2026-10-04 14:54:45 +02:00
README.md fix(lock): expose authoritative renewal lifecycle 2026-07-17 07:04:14 +02:00
build.rs waymaker-client v0.1.27 — polyglot Go / TypeScript / Rust clients 2026-06-09 19:20:48 +02:00

README.md

waymaker-client (Rust)

Official Rust client for waymaker.

[dependencies]
waymaker-client = { git = "https://git.awesomike.com/pub/waymaker-client", tag = "v0.1.31" }
# (the crate lives in the rust/ subdir; cargo resolves it automatically)

Proto stubs are generated from ../proto at build time via build.rs — no dependency on the waymaker server workspace.

Surfaces

  • lock — read/write locks, leader election, TTL leases. The Lock handle keeps a background task that transparently re-binds its event stream across a primary bounce (reusing the request_id) and exposes live state: lock.fence_token(), lock.watch() (a watch::Receiver<LockState>), lock.is_lost(). LockState::generation distinguishes a same-generation re-bind from a fresh transparent acquire, while LockState::renewal exposes typed renewal success/failure and the authoritative renewed deadline. Pin the id/token/generation before protected work and reject any change.
  • stream — JetStream-lite publish + pull/push consumers.
  • kv — Put/Get/Create/Update(CAS)/Delete/Keys/History/Watch.
  • cache — Redis-shape Hash / Set / Queue (collections).
  • probabilistic — Bloom / HLL / CMS / TopK / t-digest.
  • object — chunked object Put/Get.

Raw generated stubs are available under server / streams_server / kv_server / collections_server / sketches_server / cache_server.

Leader election

use std::time::Duration;
use waymaker_client::{Client, lock};

let client = Client::connect("http://127.0.0.1:8818").await?;
let lock = client.acquire_lock("leader:reports", lock::Config {
    max_wait: Duration::ZERO,            // try-acquire
    lease_ttl: Duration::from_secs(30),
    scope: lock::Scope::Quorum,          // Raft-replicated fence
    ..Default::default()
}).await?;
let _renewal = lock.spawn_renewal(Duration::from_secs(15));

// Pin one ownership generation; a transparent re-win cannot inherit work.
let mut state = lock.watch();
let admitted = state.borrow().clone();
loop {
    let current = state.borrow().clone();
    if current.lost
        || current.generation != admitted.generation
        || current.id != admitted.id
        || current.fence_token != admitted.fence_token
        || matches!(current.renewal, lock::RenewalState::Failed(_))
    {
        break;
    }
    do_fenced_write(admitted.fence_token).await?;
    state.changed().await.ok();
}