Mukund Tiwari

About

I build and run the infrastructure that keeps high-traffic products up and makes AI inference cheap. Over 5+ years I've worked on AWS platforms, production Kubernetes, and self-hosted GPU inference, mostly on small teams where I own things end to end.

Right now I'm a Senior DevOps Engineer at Primetrace, where I handle load testing, the in-house inference platform, and most of our AWS setup.

Selected work

• Load testing for peak events. Prepared Crafto for ~10M requests/min (~167k RPS). It held at 1.6M concurrent users and 19M+ DAU, and did INR 5.4 Cr in revenue in 48 hours.
• Self-hosted image models. Flux models I deployed now serve 30% of production traffic, bringing per-inference cost from INR 3.5–4 down to INR 0.8.
• Self-hosted LLMs. Run an 8× L40S Gemma 4 26B-A4B (MoE) fleet on vLLM (49B+ prompt tokens served so far) and Gemma 4 31B FP8 on Blackwell, cutting reliance on third-party LLM APIs.
• llm-d on EKS. Ran an end-to-end POC of llm-d's prefix- and load-aware routing (Gateway API Inference Extension, Envoy + EPP) in front of vLLM on GPU nodes. Against round-robin it gave 7× faster p50 TTFT, 2× throughput, and 74% vs 46% cache hits.
• GPU cost cuts. Moved upscaling from 2× A100 to 1× L40S (68% cheaper) and brought other upscalers in-house, removing $3,200/mo in Replicate spend.
• Autoscaling video rendering. Designed KEDA autoscaling for the SQS-driven render worker ahead of a full rollout: scheduled capacity floors plus queue-depth scaling, safe fallback caps, and Argo CD fixes, sized for ~1,230-pod festival peaks on Karpenter.
• Passwordless databases. Rolling out RDS IAM auth across Crafto services, with token and credential caching and IAM-only MySQL users over TLS 1.3.
• Real-time voice AI. Designed and shipped a WebRTC calling stack on EKS (coturn behind an NLB, pipecat/aiortc pods) with sub-second latency.
• Cloud savings at Ollion. Saved an Indonesian digital bank $230K/year through EC2 rightsizing and VPC endpoint consolidation.

Open source

• vLLM. Authored #60821, root-causing slow long-context prefill on T4 (Turing) GPUs: Triton attention runs without tensor cores there since Triton 3.3, and FlashInfer cuts 8k-token TTFT from 9.5s to 0.6s. Contributed a head-size check adopted into #55380.

Writing

• How did we load test for 100x traffic? · Medium

Experience

• Senior DevOps Engineer, PrimetraceDec 2024 – now

  Inference platform, load testing, EKS, Terraform, MySQL, and security on a two-person DevOps team.

• DevOps Engineer, OllionAug 2022 – Nov 2024

  Cost optimisation, a GKE → EKS migration, and security hardening across 80+ AWS accounts.

• Senior Systems Engineer, InfosysNov 2020 – Aug 2022

  Azure (AKS) migrations with Terraform for large banking clients.

Also

• 1st place, AWS GameDay: a live multi-team competition across startups.
• Certified: GCP Professional Cloud Architect and Cloud Security Engineer, AWS Solutions Architect Associate, Terraform Associate, Azure Administrator.
• B.Tech, Electronics & Communication, Maharaja Agrasen Institute of Technology, Delhi (2020).
• Class 12, CBSE, Apeejay School, Pitampura, Delhi (2016).