Vibeleaderboard
← All Intel
Intel / podcast

Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon

Source
lennysnewsletter.com
Author
Lenny Rachitsky
Date

Context

An AI product can earn greater autonomy through a deliberately limited first version. AI deployment specialists Aishwarya Naresh Reganti and Kiriti Badam describe a customer-support progression: initially, the system suggests replies to human staff, whose corrections reveal where it fails. Once those weaknesses are understood, a team can consider letting it answer customers directly and later perform actions such as refunds. Each stage supplies evidence for the next permission decision. Their proposed workflow makes human correction part of product development, rather than treating full autonomy as the starting requirement.

Key quotes

“You don't know how the user might behave with your product, and you also don't know how the LLM might respond to that.”

Aishwarya Naresh Reganti

“It's not about being the first company to have an agent among your competitors. It's about have you built the right fly wheels in place so that you can improve over time.”

Aishwarya Naresh Reganti

“I probably will go as far to say that if someone's selling you one click agents it's pure marketing, you don't want to buy into that.”

Aishwarya Naresh Reganti

“You must be comfortable with the fact that your intuitions might not be right, and you probably are the dumbest person in the room, and you wanna learn from everyone.”

Aishwarya Naresh Reganti

“talking to an engineer in like any random company, especially outside of the area, you can see like the amount of impact this coding agent can create and the penetration is very low.”

Kiriti Badam
Chapters
  • 00:00Introduction to Aishwarya and Kiriti
  • 05:03Challenges in AI product development
  • 07:36Key differences between AI and traditional software
  • 13:19Building AI products: start small and scale
  • 15:23The importance of human control in AI systems
  • 22:38Avoiding prompt injection and jailbreaking
  • 25:18Patterns for successful AI product development
  • 33:20The debate on evals and production monitoring
  • 41:27Codex team's approach to evals and customer feedback
  • 45:41Continuous calibration, continuous development (CC/CD) framework
  • 58:07Emerging patterns and calibration
  • 01:01:24Overhyped and under-hyped AI concepts
  • 01:05:17The future of AI
  • 01:08:41Skills and best practices for building AI products
  • 01:14:04Lightning round and final thoughts
Read the source www.lennysnewsletter.com
More from Lenny Rachitsky
Recommended reads
Comments

Checking sign-in…

Loading comments…