• transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

A rare account of running transformer inference at roughly 2 million predictions per second in production, including the graph-aware modeling choices and serving tradeoffs that made it viable.

• articleUber Cart Assistant
• articleSolving The Agent Identity Crisis
• articleUbers File Analyzer

• articleAccelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModelhuggingface.co
• articleFast Inference from Transformers via Speculative DecodingYaniv Leviathan et al.
• articleLarge Transformer Model Inference OptimizationLilian Weng

Checking sign-in…

Loading comments…