haiyang.dev

Software Engineer

Haiyang
Luo

Working on backend infrastructure and distributed systems.

GitHub LinkedIn Email

KV cache playground

Watch requests flow through the fleet, then open the playground to try it.

Inference batches routed across three GPU clusters One queue feeds three illustrative GPU clusters with different processing speeds. Each cluster contains one eight-GPU node. Each inference batch uses a one-, two-, four-, or eight-GPU replica in a single node. Scheduling considers available replica groups and local prefix KV cache.

Prefill processes the prompt; decode generates the response.

    No batches in flight.

    Simulated progress per batch. Completed batches leave the list.

    Ready · 24 GPUs available

    Same prefix. Less prefill.

    Compare two batches on a separate 1-GPU replica. Your fleet keeps running.

    Batches wait until their target node has a free GPU replica.

    Simplified fleet simulation. Prefix KV stays on the same model and GPU replica.

    How does the cache work?

    Prompt: With “Reuse prompt prefix” on, requests share a 1,024-token prefix and add 128 fresh tokens. Turn it off for a completely new prompt. A hit requires the same exact prefix, model, and GPU replica.

    Storage: Prefix KV stays in that replica’s GPU memory, sharded across its GPUs. KV stored means a node has cached prefixes. In this fleet, nothing transfers between replicas; changing a GPU group’s model configuration clears its KV. This demo retains only the shared prefix between batches.

    Reuse: Hits skip the shared prefix’s prefill. Fresh tokens still need prefill; decode still generates an answer. Hit rate = batches with a prefix hit / assigned batches.

    Controls: Route pins your batches to a cluster; they wait if it is busy or paused. Click a cluster to pause / resume it. Background demo traffic always routes automatically.

    Simple simulation: A dot is an inference batch; 1/2/4/8 GPUs represent model replica sizes. Models are assumed loaded. Exclusive placement, tokens, and timings are illustrative. No real model, continuous batching, memory limits, or memory-pressure eviction.

    Engineering

    Founding engineer at Vybers, focused on cloud platforms and AI inference infrastructure.

    Previously, I worked on Nebula at Epic Systems, building developer tooling and distributed systems.

    Open source

    LeWorldModel

    Beyond engineering

    I scuba dive and freedive.

    PADI Rescue DiverAIDA 3-star FreediverCardiopulmonary Resuscitation (CPR) certified

    Freediving personal bests

    STAStatic apnea
    DYNBDynamic with bifins
    60 m ≈ 197 ft
    CWTBConstant weight with bifins
    24 m ≈ 79 ft
    DNFDynamic without fins
    50 m ≈ 164 ft

    Say hello

    GitHub LinkedIn

    KV cache playground

    From prompt to response.