How audio-video sync is evaluated without recognizing the sound

>> oh yeah. That's a fantastic question. So, um so so sound is actually a a combination of few things. One, I'm using Atmos to to to make sure that that that the the the sound quality is high enough and is understandable. B, the model will will will will learn to learn to identify key frames. Right? And and and especially because when I feed something into the model, it's it can be just a video or it can be the video plus the prompt that generated that video. So, for example, if the prompt will say the door slammed, right? It will look for a door being slammed and and and will will will match the sound at that same frame. I Did I answer your question? >> How does the model recognize sound? >> Uh so, it's both by using um um uh Atmos, but and also to to correlate the So, for example, when it's looking at the frames, right? It's making sure that that for example, the door being slammed at frame six, frame six has a a specific timestamp. So, it's looking for that that spike in the sound at that timestamp. It doesn't know that it is that sound, but it's looking for a specific spike of sound at at that timestamp. >> What about lip syncing? >> Lip syncing is an unsolved problem yet. >> [laughter] >> One question. >> We're trying though. Yeah. Yeah, I said the the question was what about lip syncing? >> Oh, Well, wasn't me, but I guess the lip syncing answer would be interesting before I ask my question. >> Uh yeah, um as I said, it is an unsolved problem still. We're still working through it. Especially for us, some some of the characters that we're trying to do a talking head are humans, right? Which, you know, we can look at the different techniques to trying to identify the the lips, but some of them are just talking, you know, talking animations that have no real correlation between, you know, the the movement of of of the mouth and and and speech. So, unfortunately, I don't have a solution for that yet.