Frontier inference architecture

Frontier MoE models have changed the memory equation.

Birbal builds a frontier inference rack around the memory, bandwidth and compute a known model actually requires.

Explore the architecture ↓ Discuss Birbal →

Working architecture · Quantitative feasibility

The problem

Model size can set the hardware footprint before bandwidth or compute do.

Sparse MoE models keep a very large weight set resident while reading only a fraction of it on each token.

1 · Capacity requirement

~1.5–1.6TB

The full checkpoint must remain resident.

resident model footprint

one token reads →

2 · Per-token traffic

~138GB

A small active subset is read on one token.

~112 GB static path

~26 GB routed experts

at target decode × 100 TPS

3 · Bandwidth requirement

~13.8TB/s

Repeated decode creates a concentrated hot set plus a broader distributed working set.

very hot recurring traffic

lighter pressure distributed

visual scale · 1 cell ≈ 10 GB

THE MISMATCH

GPU count can be driven by how much HBM the model needs to stay resident, even when the workload uses only a fraction of the bandwidth those GPUs bring.

B200 reference

B200 baseline: capacity drives the GPU count

180 GB HBM3e · up to 8 TB/s per B200

STANDARD DGX GRANULARITY

2 × DGX B200

One 8-GPU DGX B200 has 1.44 TB HBM, so the reference model does not fit entirely in one system.

HBM capacity

52–56%

HBM bandwidth

~11%

2.88 TB installed HBM · 128 TB/s peak HBM bandwidth

Arithmetic lower bound 9 × B200 1.62 TB HBM · 72 TB/s peak HBM bandwidth · ~19% of peak bandwidth required

Capacity utilization = resident model / installed HBM Bandwidth utilization = ~13.82 TB/s weight-only requirement / peak HBM bandwidth

Reference notes · alternatives to full HBM residency

Weights can live beyond HBM.

Serving stacks can offload selected weights to CPU memory and access or prefetch them as needed. That lowers HBM pressure while making the CPU↔GPU path part of inference.

MoE routing makes the working set dynamic.

Expert selection happens per token. Caching and prefetching can help, but the routed working set varies across tokens and concurrent requests.

Coherent memory extends the hierarchy.

Grace Blackwell links CPU and GPU memory over NVLink-C2C, making a larger heterogeneous memory pool easier to use.

Sources: NVIDIA DGX B200 vLLM weight offload TensorRT MoE routing Grace unified memory

BIRBAL THESIS Keep the weight set stationary in inexpensive memory, then place enough compute beside each shard to consume it.

The design inversion

A Birbal rack is designed after the model is known.

Once the model is known, its workload becomes the specification for the rack.

MODEL CHARACTERIZATION

Resident weights~1.5–1.6 TB

Base weight bytes / token~138 GB

Active parameters~104B / token

Target decode90–100 TPS

Operator graphKDA · MLA · MoE

→

BIRBAL RACK

HIGH USEFUL OCCUPANCY · TARGET

Resident weights→tile count + memory / tile

Bytes / token × TPS→independent memory channels

Active MACs→compute / tile

Operator DAG→tile groups + fabric

Latency target→scheduler + runtime

How Birbal works

The tile is Birbal’s unit of scale.

Each tile combines an independent memory channel, a stationary weight shard and enough local compute to consume that shard.

BIRBAL TILE repeatable memory + compute cell

BOARD FABRIC

LOCAL COMPUTE CPU / GPU / NPU sized to consume the local memory stream

MEMORY INTERFACE Independent DDR channel

LOCAL MEMORY Stationary weight shard

OPTIMIZATION RULE Once local compute can consume the memory stream, the shard becomes bandwidth-bound.

Shard weights across independent memory channels.

The model stays distributed across the rack rather than being moved through a shared memory path.

Size compute to each shard.

Choose the compute engine that can keep its local memory channel busy.

Schedule the rack as one model.

Tile groups change by operator; activations and partial reductions move through the fabric while weights stay local.

ARCHITECTURAL PRINCIPLE Weights stay. Activations move.

TILES memory + compute

→

BOARDS local scheduling + reductions

→

RACK global model execution

→

API standard inference endpoint

The product

From frontier model to serving API.

01 Model model checkpoint

→

02 Characterize memory · bandwidth · compute · DAG

→

03 Configure rack tiles · fabric · runtime

→

04 Compile + map weights · operators · schedule

→

05 Serve standard inference API

Give Birbal the model. We characterize the workload, configure the rack, compile the execution plan and expose a standard serving API.

First proving workload

K3-class frontier inference.

engineering targets · full-rack performance not yet demonstrated

~2 TB installed memory target

>13 TB/s useful bandwidth requirement

90–100 TPS single-stream target

<$40k total rack BOM target

Progress

Building toward the rack.

Building the model simulator + operator DAG in progress

Scheduling simulator

Tile benchmark

Fabric prototype

Board → rack

BIRBAL

A frontier inference rack designed around the model.

Working architecture · targets under validation

Discuss Birbal →