Gelu AI

They combine three known levers — quantization, adaptive batching, and speculative decoding — but place emphasis on an integrated, purpose-built runtime. The interesting technical signal is not any single technique, it's the claim of a single engine that coordinates them together (quantized kernels + SLO-aware batching + speculative decoders) for predictable sub-second chat latency.

Adaptive batching is presented as a cost-reduction lever ("up to 60% lower cost"). That implies an SLO-aware scheduler which trades per-request latency vs. GPU utilization. This likely requires per-request metadata (priority/latency budget) and a complex queuing/scheduling policy rather than naive fixed-time batching.

Speculative decoding is highlighted as a core competitive feature but framed as quality-preserving. To do that reliably with quantized models implies a two-model pipeline (fast, lower-precision/speculative model proposing tokens + verification by the full model) and careful consistency handling (rollbacks / token acceptance), which is non-trivial when models are quantized and distributed.

Support for "custom models" and "drop-in OpenAI-compatible endpoints" together implies they have tooling to ingest arbitrary model artifacts, convert them into highly-optimized quantized formats and expose an API layer that matches OpenAI semantics (streaming, tokens, usage/billing). Packaging arbitrary weights, quantizing them safe for speculative pipelines, and ensuring API parity is a significant engineering surface.

Purpose-built engine claim suggests deep low-level work: custom CUDA/Triton kernels (or Triton alternatives), fused attention/feed-forward kernels, memory-pinned token buffers, and careful GPU memory management to run larger models on fewer GPUs. This is the sort of systems engineering that isn't obvious from marketing copy.