Harshul Jain is a senior software engineer at Audible who builds ML, data and AI infrastructure. He publishes practical material on model serving and created AgentShip to supply the infrastructure surrounding production agents.[2]
Start with LLM Inference at Scale handbook →
X · hj1393 Career
Before Audible, Jain’s professional record lists product development at DXC Technology in 2015–2017 and software engineering at Gemini Solutions in 2017–2019. His work there included OCR, operational automation and data services. He later studied computer science at NYU. These earlier roles provide context for his later focus on production infrastructure.[7]
Contributions
Connect inference choices to the bottleneck
Jain’s LLM Inference at Scale handbook separates the memory used by model weights from the key/value cache retained during generation. It connects quantization and attention architectures to model memory, and paging, batching and prefix reuse to serving behavior. That separation helps engineers choose an intervention that addresses the actual limiting resource, rather than applying every optimization to every workload. His conference workshop develops these ideas with Tanmay Sah.[5]
Supply the systems around agent logic
AgentShip packages API serving, streaming, execution and observability around agent implementations. Its repository describes adapters for Google ADK and LangGraph and integrations for tracing and evaluation. Jain presents it as a way to reuse operational machinery across agents, rather than rebuilding a service around each prompt. Deploying the framework still requires configuring its backend, model access and persistence for the intended environment.[4]
Tools, writing & talks
Tools
Start here
LLM Inference at Scale handbook
github.comAn engineering guide to model memory, cache behavior and production serving tradeoffs.
Begin with the handbook’s memory and serving sections, then compare one bottleneck with your workload. Transformer basics, Python and GPU-serving familiarity help; the guide is not a promise that every listed optimization improves every deployment.
Repository owner and author[5]
Writing, research & talks
- Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcheryoutube.com
A workshop derives LLM serving bottlenecks from first principles, showing exact KV cache memory costs, then walks through the model-side (quantization, GQA/MLA, flash attention) and serving-side (paged attention, continuous batching, prefix caching) fixes.
Catalog workOriginal source