Clip transcript
Um and they can help get you unblocked. Um so what we're going to talk about today um is really about when you start using uh a lakehouse right there's sort of two sides obviously there's a warehouse which is your structured data and your tables um and then there's the data lake part which is all of your unstructured documents and oftent times what can happen is you're given these tools like text to SQL and vector search and nowadays we don't really have trouble accessing that data Um, but sometimes there are still some challenges around how do you give your agent the right type of context, whether or not they can see all the data in the way that they need um, and really take a slice to answer the right type of question. And so what we've put together today inside of our course is an agnostic data model um that basically creates a graph representation uh both from the structured side your warehouse um but then also provides structure for some of your documents um and then allows you to do a lot of useful stuff with that. Uh we do have a scenario here that we're going to go over. Um, we've created a fictional autofix group, uh, which is, think of it like a a Pep Boys Auto or something. It's a national auto repair chain. And they have all of these bays. They have these libraries of manuals on vehicles, um, as well as safety bulletins and recalls and then a warehouse of all of the repairs that they've logged. Um, and you have sort of your floor technicians, right? These are the Danny's that are listed here where they have the cars inside of the bay and they're going to need to ask some questions. Um, you have leadership at this organization that wants to create a co-pilot uh to be able to assist these technicians on the floor. Um, and then obviously you have people like Sam who are us, the AI engineers who actually have to build the thing. Um, and so if they already have their data in a warehouse, what we're going to be looking at today is the example on BigQuery. So imagine, right, you have your documents inside of cloud storage and then um you have BigQuery as your uh data warehouse, but these patterns are also extendable to data bricks as well as snowflake. Um and essentially if they just create a co-pilot on top of uh BigQuery and the data that they have um it can pull the data correctly um but sometimes it can be confidently wrong. Um, basically it could get it could pull stuff with vector search from documents and it can do text to SQL um on just a few tables. But when those tables become massive and you get hundreds of tables or when you have large document stores where you have hundreds and thousands and millions of documents um things can get lost and fall through the cracks. And really where graph can come in to help with these shapes um is not only in these single questions that someone might have about how you know I have a broken part that I need to replace and how do I repair this vehicle but oftent times it's going to be on these estate level questions. So things like for example, what are we missing? Like what documentation maybe don't we not have to cover all of the different cars that are coming in or what documentation maybe are we not leveraging at all. So this is sort of like proving a negative which can be very hard with something like semantic search uh which can only match similar things, right? It can't really find a negative example. Um there's also questions that we have where we want patterns across everything. So say, you know, you might want to ask, well, you know, what are, you know, the common types of patterns that we see? Is there anything that we're failing to fix over and over again? Or, you know, are there, you know, specific groupings of different types of recalls that are popping up and and things of this nature where you really need to traverse the entire data set. Um, and then the other one is just asking how records relate inside of a large SQL store. Um, and when you have one big lookalike schema where you have lots of similar tables, um, how do you understand how to join those together correctly? And so there's going to be three concrete shapes that we'll introduce you to today. Uh, so the first one um, we're going to call table of contents and it's somewhat like a tree structure uh, but with also different types of links between them. You'll see how it works. Um this is going to be used on the unstructured data. Uh on the unstructured data site as well, we have something called themes. Um and what themes does is it surfers surfaces global patterns inside of your data that might not be apparent um or you might not know about beforehand. So uncovering unknown patterns and groupings. Um, and then the third one which when we take the course we're actually going to go over first is this connection shape which is essentially a semantic layer on top of your data warehouse.