← All IntelClip / EducationPersona construction can amplify bias, not reduce it
From Synthetic Personas for Market Research: Where LLM Agents Break · ≈9:59
“I am a strong liberal. I support progressive values, etc., etc.”
“In 2016, I voted for.”
“What you are going to have to do is figure it out empirically by validating against some known human ground truth data.”
What’s in it
- Explains how to prompt LLMs to simulate political personas
- Shows a real research example where persona detail backfired badly
- Gives a validation method: test synthetic personas against ground truth data
Clip transcript
techniques to kind of inspire your own synthetic personas. So, the first one is just prompting the model. Uh this right here is from the Argyle paper, which is really one of the seminal papers in this field. In fact, it's so early that the model they used was a text completion model. That's why this prompt isn't in the form of a chat. It's a statement of I am. So, they gave it a prompt that said, and for example, the middle column is basically where the context is. I am a strong liberal. I support progressive values, etc., etc. And at the end it says, "In 2016, I voted for." And they basically let the model sample its completions and it says, uh Hillary Clinton, Bernie Sanders, Hillary Clinton, and so forth. And you can see what happens with the conservative case on the top. Since this time, obviously, there've been a lot more prompting techniques. And a lot more models. And I can't tell you which prompting technique and which model is going to work best for your use case. What you are going to have to do is figure it out empirically by validating against some known human ground truth data. You'll have to do what these guys did. So, for example, here in this research, they're trying to figure out how well they can construct personas to represent voting patterns. What they found was they compared here on the left is reality and on the right is their four different types of persona constructions. And they didn't realize it at the time, but their persona construction was actually amplifying bias within the model as they got more and more detailed. And they found it was actually throwing it further and further astray from reality. So, you probably have a bunch of different ideas. The answer is you're going to have to test it and validate it against ground truth.
Comments
Checking sign-in…
Loading comments…