Secure GPU Inference Gateway
Access control and auditAn authenticated gateway in front of model endpoints. It decides who may call which model, at what budget, and what gets recorded.
01Identitycaller authenticated, model allowlist checked
02Policytoken and request budgets, rate limit, reason for access
03Routebatched, dispatched, path recorded with the response
04Executememory and capacity budgets, replayable
05Measurelatency, cost and failures to Prometheus
06Recordwho asked, what ran, what it cost
07Gatepromote, hold or roll back on regression
local current = redis.call("INCRBY", KEYS[1], ARGV[1])
if current == tonumber(ARGV[1]) then
redis.call("PEXPIRE", KEYS[1], ARGV[2])
end
if current > tonumber(ARGV[3]) then
return {0, current, tonumber(ARGV[3])}
end
return {1, current, tonumber(ARGV[3])}gateway/rate_limit.py, Redis path
Separate read, increment and expire calls can race across gateway replicas. The Lua script makes the fixed-window counter atomic, sets expiry only on the call that opens the window, and returns the decision with the updated counter. The principal id is SHA-256 hashed into the key rather than stored.
End-to-end measurement
Benchmarked against direct vLLM on the same host: Qwen2.5-3B on an RTX 5070 Ti, driven by vLLM's own serving client with paired seeds and three repetitions. 3,000 requests, no failures. Gateway throughput tracked direct within run-to-run variance, so no overhead or speedup figure is claimed.