What's Next After RLHF? — Diogo Almeida, TypeSafe AI

TypeSafe AI proposes making reliable decisions the training objective for AI embedded in software. Its presentation contrasts that goal with reinforcement learning from human feedback, which optimizes answers using human preferences. An answer people prefer and a decision dependable enough to execute automatically are different things to measure. The company says it is optimizing for calibrated decision-making, but does not disclose enough of its method or results in this talk to establish achieved reliability. The useful proposal is to define the decision a software system must make, and train toward that requirement instead of treating conversational approval as sufficient evidence.

“the simple answer is we literally put them in the loop. The goal of the loop is to optimize for human preference. It is not to run software autonomously.”

Diogo Almeida

“no matter how wrong the models are, they will look right because of the asymmetry within the reward model in RLHF”

Diogo Almeida

“What we're doing is we're just automating the writing of the software. But then it its expressibility is the same.”

Diogo Almeida

“the fact that we compress the knowledge of the internet into like this core of intelligence that then can be utilized is incredible”

Diogo Almeida

“data matters more than compute and doing the right task matters way more than data”

Diogo Almeida