Inference Stack
Run at scale with optimized model runtimes.
Geodd’s Inference Stack is the core execution layer powering our performance-driven ecosystem.
It’s built to handle large context workloads with deterministic speed, combining software level optimization and concurrency stability.
THE OPTIMIZATION LAYER
Overview
The Inference Stack is a runtime and orchestration framework designed to execute models efficiently under real-world traffic.
It eliminates common performance bottlenecks through dynamic batching, speculative decoding, and managing KV cache, ensuring predictable throughput even under load.
Our design principles
Latency Stability
p99 latency holds steady under scaling.
Throughput Efficiency
Maximum tokens-per-second-per-user, regardless of concurrency.
Hardware Utilisation
Every kernel, thread, and memory allocation is tuned for peak efficiency.
SYSTEMS DESIGN
Architecture Layers
Execution Layer
The execution layer is where model computation happens.
It includes Geodd’s Optimised Model Engine, which profiles, rewrites, and compiles models for high-throughput runtime performance.
- Dynamic graph compilation and kernel fusion
- Custom speculative decoding for 2–3× faster token generation
- Parallelised attention execution and batch concurrency optimisation
Runtime Scheduler
A concurrency-aware runtime that dynamically manages workloads based on token demand and user concurrency.
It ensures no degradation at 32+ simultaneous requests, where standard runtimes typically stall.
- Adaptive batching per request pattern
- Latency-driven scheduling with prefetching
- Custom batch scheduling
EFFICIENCY BOOST
Quantisation & Adaptive Kernels
The stack integrates a precision-aware quantisation pathway designed for efficient FP8 execution, while preserving accuracy across long-context workloads.
What this adds to the stack
Accurate FP8 quantization
With calibration-based activation scaling and optimized FP8 weight layouts.
Hybrid-precision execution
Using fused kernels for attention, MLP, and sampling where supported.
Hardware-optimised kernel selection
Choosing the most efficient kernel variants based on model dimensions and compute patterns.
Impact
DEEP HARDWARE ACCESS
CUDA-Level Extensions & Cache Manager
A set of low-level GPU optimisations designed to maximise throughput, reduce memory latency, and improve token-level predictability.
Custom CUDA Plugin Layer
Custom low-level plugin for maximum efficiency and GPU throughput.
- Custom plugins optimised for FP8/FP16 execution
- Parallel compute paths that overlap inference compute with KV-cache updates
- Pre-allocated memory pools to prevent fragmentation and allocation spikes
Accelerated Decoding Engine
Custom engine significantly accelerates generation speed while maintaining accuracy.
- Configurable integration of a draft model for accelerated decoding
- Multi-token draft generation to increase acceptance streaks
- Automatic fallback when draft sequences fail validation
- GPU-resident verification to minimise CPU–GPU round trips
LATENCY MANAGEMENT
KV Cache Router
A fast, lightweight memory layer that keeps KV-cache organised so models generate tokens quickly and consistently, even during long prompts or high traffic.
Core Features
Smart Access Patterns
Keeps the most important and recent cache segments easy to reach for faster lookups.
Cold-Block Cleanup
Tidies up or compresses cache regions that aren’t being used to save memory.
Parallel GPU Reads
Arranges cache blocks so GPU threads can read them efficiently, improving speed on long sequences.
Flexible Cache Layout
Adjusts how cache is stored and accessed when batch sizes grow or traffic spikes.
Impact
THE METRICS
Performance Highlights
OUR METHODOLOGY
Engineering Principles
Hardware-Aware Software
Every component is built with silicon-level understanding — no abstraction that wastes cycles.
Predictability First
Metrics stay stable under varying traffic conditions.
Continuous Optimisation
Stable perfoFeedback-driven tuning, not static configuration.mance at 32+ simultaneous requests
Concurrency by Design
Scaling is linear, not exponential in complexity.
Low-Level Control
Kernel fusion and memory management optimised per model type.
REAL-WORLD PROOF
Built for real workloads, not benchmarks.
Geodd’s Inference Stack combines runtime intelligence with silicon-level optimisation to deliver fast, consistent, and scalable inference — even under unpredictable traffic.
It’s the invisible engine behind every high-performance model we deploy.