The server ends a Lock stream at grant (`sender_stream` is taken), so the hold task always saw "disconnected" right after acquiring and re-acquired with the original request_id. Each re-bind got a stream that ended at once, and the Bound path had no backoff: every held lock spun at hundreds of Lock RPCs per second (~350/s measured direct, ~10k in 3s from one leptin pod at boot). On leptin prod (2026-10-04) the flood starved ExtendLease, leases lapsed by TTL (430 expiries on one node in 5h), and the spinning re-acquire re-won them under fresh ids: the holder's generation changed under it, a second pod could be granted the key meanwhile, and leptin-server — which treats a generation change as loss — crash-looped for 13h. - After the stream ends, the hold task never re-acquires. It re-proves ownership via lease_status every lease/3 (200ms..5s) and publishes `lost` when it cannot. - The server reports a missing lease as Unavailable (indistinguishable from a transport error), so `Lost` was unreachable and a lapsed lease was never reported. An unproven lease past its last proven expiry is now `lost`. - Removes the now-unused re-acquire / orphan-release path. Verified live against the leptin prod cluster through the istio mesh: two pods contending on one key for 3 min — 36 polls, 0 generation changes, mutual exclusion held; a 3s lease without renewal is reported lost at 3.0s. The Go and TS clients still use the re-bind design and are NOT fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCJdDXt4eEwJc7HHiUrM8d |
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.toml | ||
| README.md | ||
| build.rs | ||
README.md
waymaker-client (Rust)
Official Rust client for waymaker.
[dependencies]
waymaker-client = { git = "https://git.awesomike.com/pub/waymaker-client", tag = "v0.1.31" }
# (the crate lives in the rust/ subdir; cargo resolves it automatically)
Proto stubs are generated from ../proto at build time via build.rs — no
dependency on the waymaker server workspace.
Surfaces
lock— read/write locks, leader election, TTL leases. TheLockhandle keeps a background task that transparently re-binds its event stream across a primary bounce (reusing therequest_id) and exposes live state:lock.fence_token(),lock.watch()(awatch::Receiver<LockState>),lock.is_lost().LockState::generationdistinguishes a same-generation re-bind from a fresh transparent acquire, whileLockState::renewalexposes typed renewal success/failure and the authoritative renewed deadline. Pin the id/token/generation before protected work and reject any change.stream— JetStream-lite publish + pull/push consumers.kv— Put/Get/Create/Update(CAS)/Delete/Keys/History/Watch.cache— Redis-shape Hash / Set / Queue (collections).probabilistic— Bloom / HLL / CMS / TopK / t-digest.object— chunked object Put/Get.
Raw generated stubs are available under server / streams_server /
kv_server / collections_server / sketches_server / cache_server.
Leader election
use std::time::Duration;
use waymaker_client::{Client, lock};
let client = Client::connect("http://127.0.0.1:8818").await?;
let lock = client.acquire_lock("leader:reports", lock::Config {
max_wait: Duration::ZERO, // try-acquire
lease_ttl: Duration::from_secs(30),
scope: lock::Scope::Quorum, // Raft-replicated fence
..Default::default()
}).await?;
let _renewal = lock.spawn_renewal(Duration::from_secs(15));
// Pin one ownership generation; a transparent re-win cannot inherit work.
let mut state = lock.watch();
let admitted = state.borrow().clone();
loop {
let current = state.borrow().clone();
if current.lost
|| current.generation != admitted.generation
|| current.id != admitted.id
|| current.fence_token != admitted.fence_token
|| matches!(current.renewal, lock::RenewalState::Failed(_))
{
break;
}
do_fenced_write(admitted.fence_token).await?;
state.changed().await.ok();
}