AI and visualizing multidimensional vectors with Pamela Fox

There's something new from our sponsor, There's something new from our sponsor, Text Control. For decades, TXT Control's Text Control. For decades, TXT Control's Text Control. For decades, TXT Control's rendering was tied to Windows rendering was tied to Windows rendering was tied to Windows functionality, but now it's a new era. functionality, but now it's a new era. functionality, but now it's a new era. TXT text control is platform TXT text control is platform TXT text control is platform independent. You can run your .NET independent. You can run your .NET independent. You can run your .NET applications on Windows or Linux. Deploy applications on Windows or Linux. Deploy applications on Windows or Linux. Deploy with Docker containers, Kubernetes, or with Docker containers, Kubernetes, or with Docker containers, Kubernetes, or host directly in Azure App Services. host directly in Azure App Services. host directly in Azure App Services. Seamlessly integrate document editing, Seamlessly integrate document editing, Seamlessly integrate document editing, signing, collaboration, and PDF signing, collaboration, and PDF signing, collaboration, and PDF processing into your ASP.NET Core and processing into your ASP.NET Core and processing into your ASP.NET Core and Angular applications. Are you ready to Angular applications. Are you ready to Angular applications. Are you ready to see it in action? Check out the live see it in action? Check out the live see it in action? Check out the live demo at [Music] [Music] demos.extcontrol.com. Hi, I'm Scott demos.extcontrol.com. Hi, I'm Scott demos.extcontrol.com. Hi, I'm Scott Hansselman. This is another episode of Hansselman. This is another episode of Hansselman. This is another episode of Hansel Minutes. Today I'm chatting with Hansel Minutes. Today I'm chatting with Hansel Minutes. Today I'm chatting with Pamela Fox. He's a principal cloud Pamela Fox. He's a principal cloud Pamela Fox. He's a principal cloud advocate focusing on Python at advocate focusing on Python at advocate focusing on Python at Microsoft. He worked at the Con Academy Microsoft. He worked at the Con Academy Microsoft. He worked at the Con Academy and did all sorts of amazing stuff and and did all sorts of amazing stuff and and did all sorts of amazing stuff and was also a developer advocate at Google was also a developer advocate at Google was also a developer advocate at Google for a time. How are you? I'm doing for a time. How are you? I'm doing for a time. How are you? I'm doing pretty good. Just recovering from Build pretty good. Just recovering from Build pretty good. Just recovering from Build and Pyon. Yeah. Why would they put those and Pyon. Yeah. Why would they put those and Pyon. Yeah. Why would they put those two conferences near each other? What two conferences near each other? What two conferences near each other? What was that about? Well, traditionally was that about? Well, traditionally was that about? Well, traditionally Microsoft developers don't use Python, Microsoft developers don't use Python, Microsoft developers don't use Python, right? When I joined Microsoft, what right? When I joined Microsoft, what right? When I joined Microsoft, what three years ago, Python was kind of the three years ago, Python was kind of the three years ago, Python was kind of the underdog where people are like, "Oh, underdog where people are like, "Oh, underdog where people are like, "Oh, let's let's like try to get a few Python let's let's like try to get a few Python let's let's like try to get a few Python developers using Microsoft stuff." And developers using Microsoft stuff." And developers using Microsoft stuff." And then of course AI came along and now

then of course AI came along and now then of course AI came along and now Python is the first language that gets Python is the first language that gets Python is the first language that gets an SDK when we have new Microsoft an SDK when we have new Microsoft an SDK when we have new Microsoft technologies. And now .NET is like, technologies. And now .NET is like, technologies. And now .NET is like, "Hey, hey, what about .NET?" I mean, "Hey, hey, what about .NET?" I mean, "Hey, hey, what about .NET?" I mean, you're a .NET person, so you can say you're a .NET person, so you can say you're a .NET person, so you can say what it's like. I am a .NET person. I what it's like. I am a .NET person. I what it's like. I am a .NET person. I think of myself as being think of myself as being think of myself as being non-denominational, but like why can't non-denominational, but like why can't non-denominational, but like why can't we all just get along? is my we all just get along? is my we all just get along? is my perspective. But it is it is interesting perspective. But it is it is interesting perspective. But it is it is interesting how like five years ago you would have how like five years ago you would have how like five years ago you would have asked me like which language one I would asked me like which language one I would asked me like which language one I would have said JavaScript and then Python's have said JavaScript and then Python's have said JavaScript and then Python's like hey here we go you know like ML AI like hey here we go you know like ML AI like hey here we go you know like ML AI is having a moment and Python is is is having a moment and Python is is is having a moment and Python is is certainly having a number of moments. So certainly having a number of moments. So certainly having a number of moments. So I guess there's lots of languages and I guess there's lots of languages and I guess there's lots of languages and that's okay. that's okay. that's okay. Yeah, I I I I try to also be non- Yeah, I I I I try to also be non- Yeah, I I I I try to also be non- denominational when it comes to denominational when it comes to denominational when it comes to languages, uh or just generally in life languages, uh or just generally in life languages, uh or just generally in life as well, just so that there's less as well, just so that there's less as well, just so that there's less fights and and wars in the world. Um, fights and and wars in the world. Um, fights and and wars in the world. Um, but it does I do sometimes think it's so but it does I do sometimes think it's so but it does I do sometimes think it's so crazy how much time we spend rewriting crazy how much time we spend rewriting crazy how much time we spend rewriting the same things in different languages.

the same things in different languages. the same things in different languages. Like if an alien came down to Earth and Like if an alien came down to Earth and Like if an alien came down to Earth and they evaluated our state of programming, they evaluated our state of programming, they evaluated our state of programming, they'd be like, "Why do you have 20 they'd be like, "Why do you have 20 they'd be like, "Why do you have 20 different languages, right? Why do you different languages, right? Why do you different languages, right? Why do you have this proliferation? You could like have this proliferation? You could like have this proliferation? You could like as a like a society, you could be more as a like a society, you could be more as a like a society, you could be more productive if you had, you know, a productive if you had, you know, a productive if you had, you know, a single language. single language. single language. I I I have this analogy that I've taught I I I have this analogy that I've taught I I I have this analogy that I've taught I've told I've told before and it I've told I've told before and it I've told I've told before and it offended a number of people in Northern offended a number of people in Northern offended a number of people in Northern Europe, but I will use this analogy for Europe, but I will use this analogy for Europe, but I will use this analogy for you and you tell me what you think. My you and you tell me what you think. My you and you tell me what you think. My son and I were in Finland and I think he son and I were in Finland and I think he son and I were in Finland and I think he was 16 and the language, the Finnish was 16 and the language, the Finnish was 16 and the language, the Finnish language is very strange and he said, language is very strange and he said, language is very strange and he said, "Why do they bother? They're not going "Why do they bother? They're not going "Why do they bother? They're not going to win." And it was such a like a to win." And it was such a like a to win." And it was such a like a ignorant teenager thing to say, but at ignorant teenager thing to say, but at ignorant teenager thing to say, but at the same time like if you look at it the same time like if you look at it the same time like if you look at it from an alien coming to Earth from an alien coming to Earth from an alien coming to Earth perspective, it's like do we need custom perspective, it's like do we need custom perspective, it's like do we need custom languages for all these people? There's languages for all these people? There's languages for all these people? There's like like like 8,000 million people on the planet. Why 8,000 million people on the planet. Why 8,000 million people on the planet. Why do they need their own? And you know, do they need their own? And you know, do they need their own? And you know, why can't we just all speak English? And why can't we just all speak English? And why can't we just all speak English? And then I realized that JavaScript is like then I realized that JavaScript is like then I realized that JavaScript is like the English, it's just this crazy the English, it's just this crazy the English, it's just this crazy language that like colonized everything language that like colonized everything language that like colonized everything and stole from all the other languages.

and stole from all the other languages. and stole from all the other languages. And like it's not the language we And like it's not the language we And like it's not the language we deserve. It's the language that we ended deserve. It's the language that we ended deserve. It's the language that we ended up with. So like why do Finnish people up with. So like why do Finnish people up with. So like why do Finnish people bother? And it's because they have art bother? And it's because they have art bother? And it's because they have art and culture and poems and songs and you and culture and poems and songs and you and culture and poems and songs and you know plays they they want their know plays they they want their know plays they they want their language. So like why does Erlang bother language. So like why does Erlang bother language. So like why does Erlang bother when Python is winning? Why does any when Python is winning? Why does any when Python is winning? Why does any language bother people just like to not language bother people just like to not language bother people just like to not invent not invented here syndrome? They invent not invented here syndrome? They invent not invented here syndrome? They want to make their own thing. It's the want to make their own thing. It's the want to make their own thing. It's the only answer. Yeah. I mean, when it comes only answer. Yeah. I mean, when it comes only answer. Yeah. I mean, when it comes to natural languages, I will defend to natural languages, I will defend to natural languages, I will defend those like to the death because natural those like to the death because natural those like to the death because natural languages I Yeah, I I did a minor in languages I Yeah, I I did a minor in languages I Yeah, I I did a minor in linguistics and I actually like my linguistics and I actually like my linguistics and I actually like my research, you know, was around multiple research, you know, was around multiple research, you know, was around multiple languages and how they kind of connect languages and how they kind of connect languages and how they kind of connect to each other and and whatnot because to each other and and whatnot because to each other and and whatnot because it's it's it's just beautiful when you it's it's it's just beautiful when you it's it's it's just beautiful when you learn another language and like you'll learn another language and like you'll learn another language and like you'll learn like a word that exists in a learn like a word that exists in a learn like a word that exists in a language that doesn't even exist in language that doesn't even exist in language that doesn't even exist in English and you're like, "Oh, we we need English and you're like, "Oh, we we need English and you're like, "Oh, we we need that word." I know German is famous for that word." I know German is famous for that word." I know German is famous for them but like in Portuguese um they have them but like in Portuguese um they have them but like in Portuguese um they have the word like sa which is you say like the word like sa which is you say like the word like sa which is you say like oh it means like oh I'm I have this huge oh it means like oh I'm I have this huge oh it means like oh I'm I have this huge feeling of missing you right so it means feeling of missing you right so it means feeling of missing you right so it means like missing but it's it's I don't know like missing but it's it's I don't know like missing but it's it's I don't know it's got all this all other stuff it's got all this all other stuff it's got all this all other stuff wrapped up in it and it's just beautiful wrapped up in it and it's just beautiful wrapped up in it and it's just beautiful and and and um so it's like you know like having all um so it's like you know like having all um so it's like you know like having all these languages because like the you these languages because like the you these languages because like the you know the richness of human experiences know the richness of human experiences know the richness of human experiences right like I did a research project right like I did a research project right like I did a research project about spatial prepositions and in order about spatial prepositions and in order about spatial prepositions and in order I wanted to teach a computer spatial I wanted to teach a computer spatial I wanted to teach a computer spatial prepositions, uh, meaning like over, prepositions, uh, meaning like over, prepositions, uh, meaning like over, under, around, that sort of stuff, under, around, that sort of stuff, under, around, that sort of stuff, right? So, in order to figure out how a right? So, in order to figure out how a right? So, in order to figure out how a computer would learn it, I first read

computer would learn it, I first read computer would learn it, I first read like a whole textbook about spatial like a whole textbook about spatial like a whole textbook about spatial prepositions across languages to see how prepositions across languages to see how prepositions across languages to see how different languages do spatial different languages do spatial different languages do spatial prepositions because some of them will prepositions because some of them will prepositions because some of them will be like, uh, you know, we'll talk about be like, uh, you know, we'll talk about be like, uh, you know, we'll talk about like, oh, on this side of the mountain like, oh, on this side of the mountain like, oh, on this side of the mountain or this the other side of the mountain, or this the other side of the mountain, or this the other side of the mountain, right? Because that's how it made sense right? Because that's how it made sense right? Because that's how it made sense in their culture to describe places. Um, in their culture to describe places. Um, in their culture to describe places. Um, so then when I taught the computer so then when I taught the computer so then when I taught the computer spatial prepositions, I actually I used spatial prepositions, I actually I used spatial prepositions, I actually I used I think 3D Studio Max. It was a 3D I think 3D Studio Max. It was a 3D I think 3D Studio Max. It was a 3D modeling package and had it like modeling package and had it like modeling package and had it like measuring like different angles and measuring like different angles and measuring like different angles and relations to each other and then it relations to each other and then it relations to each other and then it would like form a decision tree and then would like form a decision tree and then would like form a decision tree and then learn the prepositions, right? But this learn the prepositions, right? But this learn the prepositions, right? But this like that's that I think is what's like that's that I think is what's like that's that I think is what's beautiful about languages is that we beautiful about languages is that we beautiful about languages is that we have so many different ways of viewing have so many different ways of viewing have so many different ways of viewing the world and if you learn a lon the world and if you learn a lon the world and if you learn a lon language, you can actually like change language, you can actually like change language, you can actually like change the way you view the world. the way you view the world. the way you view the world. Yeah. the the there's that whole myth Yeah. the the there's that whole myth Yeah. the the there's that whole myth about the no like Eskimos have 90 words about the no like Eskimos have 90 words about the no like Eskimos have 90 words for snow and that kind of thing. But but for snow and that kind of thing. But but for snow and that kind of thing. But but as a general rule like I've noticed in as a general rule like I've noticed in as a general rule like I've noticed in my relationship with my wife that I'm my relationship with my wife that I'm my relationship with my wife that I'm very much cardinal directions and she's very much cardinal directions and she's very much cardinal directions and she's very much relativistic. And I was very much relativistic. And I was very much relativistic. And I was surprised that like English doesn't have surprised that like English doesn't have surprised that like English doesn't have more precise ways of describing more precise ways of describing more precise ways of describing direction. For her it's very much like direction. For her it's very much like direction. For her it's very much like go to the monument, hang a left at the go to the monument, hang a left at the go to the monument, hang a left at the statue. And for me it's like head east statue. And for me it's like head east statue. And for me it's like head east 300 y, turn left 90°. too. I'm basically 300 y, turn left 90°. too. I'm basically 300 y, turn left 90°. too. I'm basically a logo robot and everything that she a logo robot and everything that she a logo robot and everything that she does as does is 100% relativistic and does as does is 100% relativistic and does as does is 100% relativistic and I'm very much absolute and I'm I'm I'm I'm very much absolute and I'm I'm I'm I'm very much absolute and I'm I'm I'm frustrated sometimes that I don't have frustrated sometimes that I don't have frustrated sometimes that I don't have the support language to to get through the support language to to get through the support language to to get through those situations.

those situations. those situations. Yeah, I'm one of those people if you Yeah, I'm one of those people if you Yeah, I'm one of those people if you tell me to go north, I'll just go uphill tell me to go north, I'll just go uphill tell me to go north, I'll just go uphill cuz I just think our big argument is cuz I just think our big argument is cuz I just think our big argument is which way the GPS map is like oriented. which way the GPS map is like oriented. which way the GPS map is like oriented. For her, the earth has to rotate around For her, the earth has to rotate around For her, the earth has to rotate around her and for me north is vertical. like her and for me north is vertical. like her and for me north is vertical. like north is up always. So if you have a north is up always. So if you have a north is up always. So if you have a background in linguistic then background in linguistic then background in linguistic then linguistics did you see AI and things linguistics did you see AI and things linguistics did you see AI and things like vector embeddings coming or did you like vector embeddings coming or did you like vector embeddings coming or did you just happen to study the perfect thing just happen to study the perfect thing just happen to study the perfect thing for your your chosen work? for your your chosen work? for your your chosen work? Uh I didn't I wasn't like up to date Uh I didn't I wasn't like up to date Uh I didn't I wasn't like up to date with linguistics research like I didn't with linguistics research like I didn't with linguistics research like I didn't I didn't do a PhD or anything. It was I didn't do a PhD or anything. It was I didn't do a PhD or anything. It was just a minor. just a minor. just a minor. Um, but I was I was happy when you know Um, but I was I was happy when you know Um, but I was I was happy when you know I am happy that LLMs are a thing now I am happy that LLMs are a thing now I am happy that LLMs are a thing now because I feel like I can finally bring because I feel like I can finally bring because I feel like I can finally bring in my you know my language background. in my you know my language background. in my you know my language background. Um cuz traditionally like computer Um cuz traditionally like computer Um cuz traditionally like computer science wasn't that associated with science wasn't that associated with science wasn't that associated with language like if you were applying to a language like if you were applying to a language like if you were applying to a computer science camp or course they computer science camp or course they computer science camp or course they would look at your math scores right for would look at your math scores right for would look at your math scores right for the SAT right like there was a C I went the SAT right like there was a C I went the SAT right like there was a C I went to computer camp when I was a kid and um to computer camp when I was a kid and um to computer camp when I was a kid and um in order to qualify to go to the in order to qualify to go to the in order to qualify to go to the computer camp they would look at the computer camp they would look at the computer camp they would look at the your like SAT score um and they would your like SAT score um and they would your like SAT score um and they would only look at the math part because they only look at the math part because they only look at the math part because they thought like oh math computer science thought like oh math computer science thought like oh math computer science right so there's always been this like right so there's always been this like right so there's always been this like like you know connection between math like you know connection between math like you know connection between math and computer science science there and computer science science there and computer science science there hasn't been that strong of connection hasn't been that strong of connection hasn't been that strong of connection between language and computer science between language and computer science between language and computer science but even in college I was actually doing but even in college I was actually doing but even in college I was actually doing I did my compilers class at the same I did my compilers class at the same I did my compilers class at the same time as my uh linguistics syntax class

time as my uh linguistics syntax class time as my uh linguistics syntax class so I was literally writing parse trees so I was literally writing parse trees so I was literally writing parse trees for computer programs at the same time for computer programs at the same time for computer programs at the same time that I was writing parse trees for like that I was writing parse trees for like that I was writing parse trees for like English and Japanese sentences right and English and Japanese sentences right and English and Japanese sentences right and I ended up doing so well in both those I ended up doing so well in both those I ended up doing so well in both those classes that like I like ended I set the classes that like I like ended I set the classes that like I like ended I set the curve yada yada because But because like curve yada yada because But because like curve yada yada because But because like I was doing parse trees everywhere and I was doing parse trees everywhere and I was doing parse trees everywhere and my whole life was parse trees at that my whole life was parse trees at that my whole life was parse trees at that point and everything just clicked and point and everything just clicked and point and everything just clicked and made sense and I could see the world as made sense and I could see the world as made sense and I could see the world as pars trees you know it was like the pars trees you know it was like the pars trees you know it was like the matrix and uh and yeah so I like think matrix and uh and yeah so I like think matrix and uh and yeah so I like think from an early early stage I've seen the from an early early stage I've seen the from an early early stage I've seen the connections between um computer science connections between um computer science connections between um computer science and language and it's very convenient and language and it's very convenient and language and it's very convenient that now I get to bring more language that now I get to bring more language that now I get to bring more language into computer programming. It sets you into computer programming. It sets you into computer programming. It sets you up for success 100%. up for success 100%. up for success 100%. Yeah. Yeah. I mean, it helps. I There's Yeah. Yeah. I mean, it helps. I There's Yeah. Yeah. I mean, it helps. I There's still a ton of uh math involved in LLM. still a ton of uh math involved in LLM. still a ton of uh math involved in LLM. So, now I'm I am wishing that I paid So, now I'm I am wishing that I paid So, now I'm I am wishing that I paid more attention in the linear algebra more attention in the linear algebra more attention in the linear algebra class in college. The thing is during class in college. The thing is during class in college. The thing is during the class they didn't say what linear the class they didn't say what linear the class they didn't say what linear algebra was used for. They didn't say algebra was used for. They didn't say algebra was used for. They didn't say like, "Hey, by the way, like linear like, "Hey, by the way, like linear like, "Hey, by the way, like linear algebra is the basis of computer algebra is the basis of computer algebra is the basis of computer graphics renderers. Linear algebra is, graphics renderers. Linear algebra is, graphics renderers. Linear algebra is, you know, in the future going to be used you know, in the future going to be used you know, in the future going to be used for right at the time." You know, I for right at the time." You know, I for right at the time." You know, I didn't realize how useful that class didn't realize how useful that class didn't realize how useful that class would be. So, I do wish I had a bit more would be. So, I do wish I had a bit more would be. So, I do wish I had a bit more linear algebra um expertise. Uh but I linear algebra um expertise. Uh but I linear algebra um expertise. Uh but I have been trying to make up for that by have been trying to make up for that by have been trying to make up for that by reading lots of books that do lots of reading lots of books that do lots of reading lots of books that do lots of matrix math. Mhm. Do me a favor and matrix math. Mhm. Do me a favor and matrix math. Mhm. Do me a favor and let's try to put our professor hats on let's try to put our professor hats on let's try to put our professor hats on and talk about vector embeddings in a

and talk about vector embeddings in a and talk about vector embeddings in a way that doesn't float away like a way that doesn't float away like a way that doesn't float away like a helium balloon and get up into the like helium balloon and get up into the like helium balloon and get up into the like high complexity of it cuz I think high complexity of it cuz I think high complexity of it cuz I think analogies are a valuable way for us as analogies are a valuable way for us as analogies are a valuable way for us as for myself rather as a lay person and for myself rather as a lay person and for myself rather as a lay person and perhaps folks that are listening because perhaps folks that are listening because perhaps folks that are listening because I feel like when we have conversations I feel like when we have conversations I feel like when we have conversations about how LLMs work and how they think about how LLMs work and how they think about how LLMs work and how they think and I put think in in quotes um it it and I put think in in quotes um it it and I put think in in quotes um it it floats very very quickly. You can have a floats very very quickly. You can have a floats very very quickly. You can have a you know it's kind of like a parrot and you know it's kind of like a parrot and you know it's kind of like a parrot and then we go and here's the vector then we go and here's the vector then we go and here's the vector embedding and here's the math and it's embedding and here's the math and it's embedding and here's the math and it's like whoa whoa and there's no space like whoa whoa and there's no space like whoa whoa and there's no space between the two. I want to understand between the two. I want to understand between the two. I want to understand the idea that a vector embedding is a the idea that a vector embedding is a the idea that a vector embedding is a way to represent way to represent way to represent relationships. Is there help me relationships. Is there help me relationships. Is there help me understand that so embeddings are just understand that so embeddings are just understand that so embeddings are just numbers? Where's the magic there? numbers? Where's the magic there? numbers? Where's the magic there? So yeah, the vector embedding well it's So yeah, the vector embedding well it's So yeah, the vector embedding well it's a vector in multi-dimensional space. So a vector in multi-dimensional space. So a vector in multi-dimensional space. So you know we can't visualize thousands of you know we can't visualize thousands of you know we can't visualize thousands of dimensions but I just always think of it dimensions but I just always think of it dimensions but I just always think of it as an ice cream cone. Uh so uh you know as an ice cream cone. Uh so uh you know as an ice cream cone. Uh so uh you know imagine your ice cream cone full of imagine your ice cream cone full of imagine your ice cream cone full of vectors right um and uh you know you've vectors right um and uh you know you've vectors right um and uh you know you've got a vector for dog and you've got the got a vector for dog and you've got the got a vector for dog and you've got the vector for cat and those vectors are vector for cat and those vectors are vector for cat and those vectors are quite close to each other because a lot quite close to each other because a lot quite close to each other because a lot of times cat and dog would be used in of times cat and dog would be used in of times cat and dog would be used in very similar situations right so the dog very similar situations right so the dog very similar situations right so the dog sat on the mat the cat sat on the mat sat on the mat the cat sat on the mat sat on the mat the cat sat on the mat they can both sit on the mat I have a they can both sit on the mat I have a they can both sit on the mat I have a pet dog I have a pet cat right they have pet dog I have a pet cat right they have pet dog I have a pet cat right they have a huge amount of semantic similarity a huge amount of semantic similarity a huge amount of semantic similarity right um so they you know those two right um so they you know those two right um so they you know those two vectors would be, you know, closer to to

vectors would be, you know, closer to to vectors would be, you know, closer to to each other in this um in this vector each other in this um in this vector each other in this um in this vector space. When you say closer to each space. When you say closer to each space. When you say closer to each other, are they head like I'm imagining other, are they head like I'm imagining other, are they head like I'm imagining arrows heading in the generally same arrows heading in the generally same arrows heading in the generally same direction in the kind of like small direction in the kind of like small direction in the kind of like small furry things with four feet direction? furry things with four feet direction? furry things with four feet direction? Is that what we're trying to say? Yeah. Is that what we're trying to say? Yeah. Is that what we're trying to say? Yeah. The tricky thing with the vector The tricky thing with the vector The tricky thing with the vector embeddings is that we, you know, we embeddings is that we, you know, we embeddings is that we, you know, we can't like actually say that a can't like actually say that a can't like actually say that a particular dimension means small furry particular dimension means small furry particular dimension means small furry things because we've turned this very, things because we've turned this very, things because we've turned this very, you know, multi-dimensional world you know, multi-dimensional world you know, multi-dimensional world because if you think of all the possible because if you think of all the possible because if you think of all the possible ways you could describe something, ways you could describe something, ways you could describe something, there'd be like millions of ways to there'd be like millions of ways to there'd be like millions of ways to describe something be like things you describe something be like things you describe something be like things you can own, things you can pet, things you can own, things you can pet, things you can own, things you can pet, things you can feed, things that are, you know, can feed, things that are, you know, can feed, things that are, you know, canines, right? There's there's too many canines, right? There's there's too many canines, right? There's there's too many dimensions, right? So those are they dimensions, right? So those are they dimensions, right? So those are they they've been reduced down. Um but yeah, they've been reduced down. Um but yeah, they've been reduced down. Um but yeah, you could generally think of them as you could generally think of them as you could generally think of them as like okay that they're like pointed in like okay that they're like pointed in like okay that they're like pointed in the same direction of furry things we the same direction of furry things we the same direction of furry things we can pet but who decides can pet but who decides can pet but who decides right we've we've abstracted it. So the right we've we've abstracted it. So the right we've we've abstracted it. So the decisions are not made absolutely the decisions are not made absolutely the decisions are not made absolutely the decisions are made relatively through decisions are made relatively through decisions are made relatively through the just creation of a corpus of text.

the just creation of a corpus of text. the just creation of a corpus of text. So when someone makes a a a Ghostbusters So when someone makes a a a Ghostbusters So when someone makes a a a Ghostbusters quote, cats and dogs living together, quote, cats and dogs living together, quote, cats and dogs living together, mass hysteria, which is a very mass hysteria, which is a very mass hysteria, which is a very generational quote, like you either grew generational quote, like you either grew generational quote, like you either grew up and you you know that quote or you're up and you you know that quote or you're up and you you know that quote or you're under 30 and you've never heard that under 30 and you've never heard that under 30 and you've never heard that quote. That text appears on the quote. That text appears on the quote. That text appears on the internet millions of times because of internet millions of times because of internet millions of times because of Ghostbusters, not because of anything Ghostbusters, not because of anything Ghostbusters, not because of anything specific to dogs and cats, but cats and specific to dogs and cats, but cats and specific to dogs and cats, but cats and dogs, dogs and cats, those words are dogs, dogs and cats, those words are dogs, dogs and cats, those words are always near each other. Does their always near each other. Does their always near each other. Does their proximity then cause, for lack of a proximity then cause, for lack of a proximity then cause, for lack of a better word, better word, better word, bias where dogs and cats are always bias where dogs and cats are always bias where dogs and cats are always together, but like hamsters and ferrets together, but like hamsters and ferrets together, but like hamsters and ferrets are like getting the the short end of are like getting the the short end of are like getting the the short end of the stick when it comes to like furry the stick when it comes to like furry the stick when it comes to like furry things. Uh, that's a great question. It things. Uh, that's a great question. It things. Uh, that's a great question. It is going to be based off of what Well, is going to be based off of what Well, is going to be based off of what Well, it's going to be based on both the data it's going to be based on both the data it's going to be based on both the data that the embedding model was trained on that the embedding model was trained on that the embedding model was trained on and just the way in which they like fed and just the way in which they like fed and just the way in which they like fed that data into the system, right? So for that data into the system, right? So for that data into the system, right? So for example like if you want to you can make example like if you want to you can make example like if you want to you can make your own embedding model using the your own embedding model using the your own embedding model using the wordtovec algorithm uh and just feed in wordtovec algorithm uh and just feed in wordtovec algorithm uh and just feed in whatever data set like I did it using whatever data set like I did it using whatever data set like I did it using the Google news data set. So that would the Google news data set. So that would the Google news data set. So that would you know that would tend to you know you know that would tend to you know you know that would tend to you know base its idea of relationships based off base its idea of relationships based off base its idea of relationships based off of what it sees in in Google News. Um, of what it sees in in Google News. Um, of what it sees in in Google News. Um, most of the these embedding models, the most of the these embedding models, the most of the these embedding models, the ones we're using these days, like the ones we're using these days, like the ones we're using these days, like the OpenAI text embedding 3 uh series, those OpenAI text embedding 3 uh series, those OpenAI text embedding 3 uh series, those are, you know, generally based off the are, you know, generally based off the are, you know, generally based off the the whole internet, right? So, um, you the whole internet, right? So, um, you the whole internet, right? So, um, you know, they may have some bias from that know, they may have some bias from that know, they may have some bias from that Ghostbuster quote, but I think there's a Ghostbuster quote, but I think there's a Ghostbuster quote, but I think there's a lot more cats and dogs out there. And I lot more cats and dogs out there. And I lot more cats and dogs out there. And I think I think I think they I think

think I think I think they I think think I think I think they I think you'll also find hamsters and ferrets do you'll also find hamsters and ferrets do you'll also find hamsters and ferrets do show up together in other places like show up together in other places like show up together in other places like on, you know, pet websites and stuff. on, you know, pet websites and stuff. on, you know, pet websites and stuff. So, um, yeah. And and also they did find So, um, yeah. And and also they did find So, um, yeah. And and also they did find like you know when they found like bias like you know when they found like bias like you know when they found like bias in like some of the early GPD models in like some of the early GPD models in like some of the early GPD models especially from like Reddit like there especially from like Reddit like there especially from like Reddit like there was a famous Reddit user that had his was a famous Reddit user that had his was a famous Reddit user that had his own token. A token is like a unit of own token. A token is like a unit of own token. A token is like a unit of measurement that an LLM learns as it measurement that an LLM learns as it measurement that an LLM learns as it sees something a lot. So, this one sees something a lot. So, this one sees something a lot. So, this one Reddit user had its own token for their Reddit user had its own token for their Reddit user had its own token for their like crazy username just because they like crazy username just because they like crazy username just because they trained on the Reddit data and they trained on the Reddit data and they trained on the Reddit data and they accidentally like learned too much like accidentally like learned too much like accidentally like learned too much like they put too much importance on on what they put too much importance on on what they put too much importance on on what they saw in that Reddit data. Um, so I they saw in that Reddit data. Um, so I they saw in that Reddit data. Um, so I think that they, you know, that we've think that they, you know, that we've think that they, you know, that we've learned things from training these learned things from training these learned things from training these models that we we do need to clean up models that we we do need to clean up models that we we do need to clean up the data more. So I think the models the data more. So I think the models the data more. So I think the models these days do try to do um do have these days do try to do um do have these days do try to do um do have cleaner data or at least they do a pass cleaner data or at least they do a pass cleaner data or at least they do a pass with cleaner data. Sometimes you like with cleaner data. Sometimes you like with cleaner data. Sometimes you like start off with the messy data like the start off with the messy data like the start off with the messy data like the whole internet in order just to teach it whole internet in order just to teach it whole internet in order just to teach it how to speak languages and then you go how to speak languages and then you go how to speak languages and then you go back and you do another pass with nice back and you do another pass with nice back and you do another pass with nice data. It's like okay you learned how to data. It's like okay you learned how to data. It's like okay you learned how to speak according to the internet and speak according to the internet and speak according to the internet and Reddit and all this stuff but this is Reddit and all this stuff but this is Reddit and all this stuff but this is how we like actually want you to speak how we like actually want you to speak how we like actually want you to speak right that was and they might do right that was and they might do right that was and they might do something with uh embedding models. I I something with uh embedding models. I I something with uh embedding models. I I don't know exactly what approach they don't know exactly what approach they don't know exactly what approach they use for embedding models. Yeah. But it's use for embedding models. Yeah. But it's use for embedding models. Yeah. But it's an interesting thing to talk about an interesting thing to talk about an interesting thing to talk about because you were saying the nice data because you were saying the nice data because you were saying the nice data versus like the messier data. I was versus like the messier data. I was versus like the messier data. I was using I was in my mind as I was using I was in my mind as I was using I was in my mind as I was formulating the the question was there

formulating the the question was there formulating the the question was there is what I would call formal data like a is what I would call formal data like a is what I would call formal data like a book that was published that was like book that was published that was like book that was published that was like thought about and intentionally created thought about and intentionally created thought about and intentionally created sold edited and made the New York Times. sold edited and made the New York Times. sold edited and made the New York Times. And then there's like posting, And then there's like posting, And then there's like posting, right? And like is I don't we don't know right? And like is I don't we don't know right? And like is I don't we don't know just like we don't know how much dark just like we don't know how much dark just like we don't know how much dark matter there is and the space between matter there is and the space between matter there is and the space between the stars. I don't know the difference the stars. I don't know the difference the stars. I don't know the difference between formal appropriate scholastic between formal appropriate scholastic between formal appropriate scholastic text versus just like tweets and if the text versus just like tweets and if the text versus just like tweets and if the internet is 90% just userenerated crap internet is 90% just userenerated crap internet is 90% just userenerated crap and 10% books or vice versa. And I'm and 10% books or vice versa. And I'm and 10% books or vice versa. And I'm wondering uh if there's any way for us wondering uh if there's any way for us wondering uh if there's any way for us to have a sense of what the LLM has to have a sense of what the LLM has to have a sense of what the LLM has learned from. learned from. learned from. Well, generally when a new LLM comes Well, generally when a new LLM comes Well, generally when a new LLM comes out, we can look at the model card and out, we can look at the model card and out, we can look at the model card and the model card describes the training the model card describes the training the model card describes the training process and also describes it. Sometimes process and also describes it. Sometimes process and also describes it. Sometimes it'll describe the training data. Um it it'll describe the training data. Um it it'll describe the training data. Um it just depends on whether you know it's just depends on whether you know it's just depends on whether you know it's one of the more open models or doesn't one of the more open models or doesn't one of the more open models or doesn't depend on the name of the company. Open depend on the name of the company. Open depend on the name of the company. Open weights.

weights. weights. Uh open no open weights means you can Uh open no open weights means you can Uh open no open weights means you can actually run it locally. But an open if actually run it locally. But an open if actually run it locally. But an open if you have open weights, it doesn't you have open weights, it doesn't you have open weights, it doesn't necessarily mean that you know uh what necessarily mean that you know uh what necessarily mean that you know uh what it was trained on. So you could have it was trained on. So you could have it was trained on. So you could have open weights and still not have any idea open weights and still not have any idea open weights and still not have any idea what it was trained on. Uh but the nice what it was trained on. Uh but the nice what it was trained on. Uh but the nice thing about open weights is it does mean thing about open weights is it does mean thing about open weights is it does mean you can you know you can run it uh run you can you know you can run it uh run you can you know you can run it uh run it locally as long as you know the it locally as long as you know the it locally as long as you know the weights plus the like the algorithm to weights plus the like the algorithm to weights plus the like the algorithm to use. Um but yeah, knowing what it was use. Um but yeah, knowing what it was use. Um but yeah, knowing what it was trained on, for example, like I was trained on, for example, like I was trained on, for example, like I was looking at a um an open embedding model, looking at a um an open embedding model, looking at a um an open embedding model, Nomic, and they have a new version that Nomic, and they have a new version that Nomic, and they have a new version that just came out recently. And so this is just came out recently. And so this is just came out recently. And so this is an embedding model you can use locally an embedding model you can use locally an embedding model you can use locally with like with like with like Olama and they specifically were trying Olama and they specifically were trying Olama and they specifically were trying to make it work well across different to make it work well across different to make it work well across different languages like English, Spanish, Korean, languages like English, Spanish, Korean, languages like English, Spanish, Korean, Hindi, like all that. And so in their Hindi, like all that. And so in their Hindi, like all that. And so in their training data, they tell you what training data, they tell you what training data, they tell you what percentage of like exactly how many percentage of like exactly how many percentage of like exactly how many tokens. That's what they do is they say tokens. That's what they do is they say tokens. That's what they do is they say this is how many tokens came from Korean this is how many tokens came from Korean this is how many tokens came from Korean and this is how many tokens came from and this is how many tokens came from and this is how many tokens came from Spanish. So then at least you get some Spanish. So then at least you get some Spanish. So then at least you get some transparency into uh into the language transparency into uh into the language transparency into uh into the language distribution and sometimes they'll also distribution and sometimes they'll also distribution and sometimes they'll also say where it came from like oh it cames say where it came from like oh it cames say where it came from like oh it cames from the the books corpus or it cames from the the books corpus or it cames from the the books corpus or it cames from this corpus right and then you from this corpus right and then you from this corpus right and then you could get a a feel for what you know could get a a feel for what you know could get a a feel for what you know what what it learned what what it learned what what it learned from. Hey friends, it's time to pay the from. Hey friends, it's time to pay the from. Hey friends, it's time to pay the rent. We're going to take a pause here rent. We're going to take a pause here rent. We're going to take a pause here and have a chat with our new sponsor and have a chat with our new sponsor and have a chat with our new sponsor Sharegate by Work. I've got Shaylen Gimy Sharegate by Work. I've got Shaylen Gimy Sharegate by Work. I've got Shaylen Gimy on the line. You know, Shaylen, if you on the line. You know, Shaylen, if you on the line. You know, Shaylen, if you had 30 seconds to explain why ShareGate had 30 seconds to explain why ShareGate had 30 seconds to explain why ShareGate makes IT teams lives easier, what would makes IT teams lives easier, what would makes IT teams lives easier, what would you say? All right, start the clock. Uh,

you say? All right, start the clock. Uh, you say? All right, start the clock. Uh, everything IT teams need to manage everything IT teams need to manage everything IT teams need to manage Microsoft 365 comes out of the box. No Microsoft 365 comes out of the box. No Microsoft 365 comes out of the box. No manual needed. Flat, transparent manual needed. Flat, transparent manual needed. Flat, transparent pricing, no data caps, no per user fees. pricing, no data caps, no per user fees. pricing, no data caps, no per user fees. From planning migrations to keeping your From planning migrations to keeping your From planning migrations to keeping your environment clean and secure, we take environment clean and secure, we take environment clean and secure, we take the heavy lifting off it's shoulders. the heavy lifting off it's shoulders. the heavy lifting off it's shoulders. You know, and when you need help, you You know, and when you need help, you You know, and when you need help, you get real support from real experts. get real support from real experts. get real support from real experts. Sharegate makes Microsoft 365 management Sharegate makes Microsoft 365 management Sharegate makes Microsoft 365 management simple so IT teams can focus on what simple so IT teams can focus on what simple so IT teams can focus on what really matters. That was exactly 29 really matters. That was exactly 29 really matters. That was exactly 29 seconds. That was impressive. Very, very seconds. That was impressive. Very, very seconds. That was impressive. Very, very impressive. Just as is Sharegate by impressive. Just as is Sharegate by impressive. Just as is Sharegate by WorkLE. You can check them out at WorkLE. You can check them out at WorkLE. You can check them out at sharegate.com. It's one tool to migrate sharegate.com. It's one tool to migrate sharegate.com. It's one tool to migrate faster and secure your tenant. It's an faster and secure your tenant. It's an faster and secure your tenant. It's an out of the box Microsoft 365 migration out of the box Microsoft 365 migration out of the box Microsoft 365 migration solution. You can check them out solution. You can check them out solution. You can check them out sharegate.com and we thank them for sharegate.com and we thank them for sharegate.com and we thank them for being a sponsor of Hansel being a sponsor of Hansel being a sponsor of Hansel minutes. While we know that everything's minutes. While we know that everything's minutes. While we know that everything's biased towards English because there's biased towards English because there's biased towards English because there's so much English, but there's also like a so much English, but there's also like a so much English, but there's also like a lot of Chinese and a lot of Arabic. Are lot of Chinese and a lot of Arabic. Are lot of Chinese and a lot of Arabic. Are is it good for a model to be is it good for a model to be is it good for a model to be multilingual or is it best for it to multilingual or is it best for it to multilingual or is it best for it to focus on on a language? Oh, that's a focus on on a language? Oh, that's a focus on on a language? Oh, that's a good question. I would say that it it good question. I would say that it it good question. I would say that it it depends on your use case, right? Like if depends on your use case, right? Like if depends on your use case, right? Like if you're if you're building an app and you're if you're building an app and you're if you're building an app and your customers are only speaking a your customers are only speaking a your customers are only speaking a certain language then uh you know like certain language then uh you know like certain language then uh you know like if you're really specifically targeting if you're really specifically targeting if you're really specifically targeting you know like the Chinese audience and you know like the Chinese audience and you know like the Chinese audience and and they're you know very strictly using and they're you know very strictly using and they're you know very strictly using Chinese with your app then you probably Chinese with your app then you probably Chinese with your app then you probably want one you know you would want to pick want one you know you would want to pick want one you know you would want to pick one that's really well suited for that one that's really well suited for that one that's really well suited for that language. Um, in in the US, I really

language. Um, in in the US, I really language. Um, in in the US, I really like to use the multilingual LLMs be like to use the multilingual LLMs be like to use the multilingual LLMs be because they uh, you know, because in because they uh, you know, because in because they uh, you know, because in the US, US is is well, we're supposed to the US, US is is well, we're supposed to the US, US is is well, we're supposed to be a melting pot and we have people who be a melting pot and we have people who be a melting pot and we have people who speak all kind language. Like in speak all kind language. Like in speak all kind language. Like in California, like 50% of the people here California, like 50% of the people here California, like 50% of the people here speak Spanish, which I think is awesome speak Spanish, which I think is awesome speak Spanish, which I think is awesome because I just walk around and get to because I just walk around and get to because I just walk around and get to like speak Spanish all day. And uh and like speak Spanish all day. And uh and like speak Spanish all day. And uh and so and people love then that the fact so and people love then that the fact so and people love then that the fact that we can you know build an app and that we can you know build an app and that we can you know build an app and then they can deploy it and then their then they can deploy it and then their then they can deploy it and then their employees can type a question in Spanish employees can type a question in Spanish employees can type a question in Spanish in it. They can type a question in in it. They can type a question in in it. They can type a question in English and they can get back responses English and they can get back responses English and they can get back responses either way. Right. So um yeah. So I like either way. Right. So um yeah. So I like either way. Right. So um yeah. So I like to find ones that you know are are doing to find ones that you know are are doing to find ones that you know are are doing a good job across the main languages a good job across the main languages a good job across the main languages that uh you know users might be using. that uh you know users might be using. that uh you know users might be using. Um but uh I definitely think it's Um but uh I definitely think it's Um but uh I definitely think it's important that people are building important that people are building important that people are building models that work really well for models that work really well for models that work really well for languages that uh otherwise aren't you languages that uh otherwise aren't you languages that uh otherwise aren't you know emphasized in the training data know emphasized in the training data know emphasized in the training data because we have this big risk of because we have this big risk of because we have this big risk of becoming English ccentric you know becoming English ccentric you know becoming English ccentric you know programming kind of started off English programming kind of started off English programming kind of started off English centric for whatever reason and uh and centric for whatever reason and uh and centric for whatever reason and uh and it would be a shame it would be a shame it would be a shame to to miss out on all those beautiful to to miss out on all those beautiful to to miss out on all those beautiful languages because we all have such we languages because we all have such we languages because we all have such we don't want to disrespect Finland.

don't want to disrespect Finland. don't want to disrespect Finland. I was I was meeting with someone at the I was I was meeting with someone at the I was I was meeting with someone at the University of Venda in South Africa and University of Venda in South Africa and University of Venda in South Africa and they speak to Venda and there's like they speak to Venda and there's like they speak to Venda and there's like maybe a million people maybe two or maybe a million people maybe two or maybe a million people maybe two or three um it's kind of in South Africa three um it's kind of in South Africa three um it's kind of in South Africa and parts of Zimbabwe and he wanted to and parts of Zimbabwe and he wanted to and parts of Zimbabwe and he wanted to know when co-pilot would be in Venda and know when co-pilot would be in Venda and know when co-pilot would be in Venda and it's like you know there's like 8,000 it's like you know there's like 8,000 it's like you know there's like 8,000 million people 8 billion plus people and million people 8 billion plus people and million people 8 billion plus people and this language is like a million maybe this language is like a million maybe this language is like a million maybe two or three like is there enough text? two or three like is there enough text? two or three like is there enough text? And I'm just picking Venda as an And I'm just picking Venda as an And I'm just picking Venda as an example. It could be uh an indigenous example. It could be uh an indigenous example. It could be uh an indigenous North American language. It could be North American language. It could be North American language. It could be anything like these languages that have anything like these languages that have anything like these languages that have a million people as a general rule. Is a million people as a general rule. Is a million people as a general rule. Is it the number of people that speak it or it the number of people that speak it or it the number of people that speak it or is it the number is it the amount of is it the number is it the amount of is it the number is it the amount of text that can be publicly found? because text that can be publicly found? because text that can be publicly found? because there's also a lot of conversations there's also a lot of conversations there's also a lot of conversations about if you're going to have like about if you're going to have like about if you're going to have like indigenous indigenous indigenous uh Native American models, who owns it uh Native American models, who owns it uh Native American models, who owns it and and who controls the bias because and and who controls the bias because and and who controls the bias because you want to bias it in a way that makes you want to bias it in a way that makes you want to bias it in a way that makes sense to the culture. Um it seems like sense to the culture. Um it seems like sense to the culture. Um it seems like we have we're certainly fine when it we have we're certainly fine when it we have we're certainly fine when it comes to English like there's enough comes to English like there's enough comes to English like there's enough text, but like is there a sense of how text, but like is there a sense of how text, but like is there a sense of how much text needs to exist before you can much text needs to exist before you can much text needs to exist before you can have a useful have a useful have a useful chatbot in a language? Yeah. So I'm just chatbot in a language? Yeah. So I'm just chatbot in a language? Yeah. So I'm just looking at the the nomic embed uh embed looking at the the nomic embed uh embed looking at the the nomic embed uh embed model, right? And so for example for model, right? And so for example for model, right? And so for example for English they bring in English they bring in English they bring in 234 is that million? Yeah.

234 is that million? Yeah. 234 is that million? Yeah. 234 million 234 million 234 million um examples. Uh so pairs because this is um examples. Uh so pairs because this is um examples. Uh so pairs because this is for an embedding model, right? So 234 for an embedding model, right? So 234 for an embedding model, right? So 234 million pairs versus uh like IGBO. Uh, million pairs versus uh like IGBO. Uh, million pairs versus uh like IGBO. Uh, Igbbo. That's a language in Igbo is Igbbo. That's a language in Igbo is Igbbo. That's a language in Igbo is Nigerian in Nigeria. Yeah. And so that Nigerian in Nigeria. Yeah. And so that Nigerian in Nigeria. Yeah. And so that that one Igbo. Okay. No, I think the G that one Igbo. Okay. No, I think the G that one Igbo. Okay. No, I think the G is is is silent. Oh, you don't even say is is is silent. Oh, you don't even say is is is silent. Oh, you don't even say the G. Okay. I It's like an I mean I the G. Okay. I It's like an I mean I the G. Okay. I It's like an I mean I could be wrong. Neither of us was could be wrong. Neither of us was could be wrong. Neither of us was Nigerian, but it's kind of like the G is Nigerian, but it's kind of like the G is Nigerian, but it's kind of like the G is kind of like it's there like cookies and kind of like it's there like cookies and kind of like it's there like cookies and cream. It's just kind of like I But I cream. It's just kind of like I But I cream. It's just kind of like I But I could be wrong. We're going to get could be wrong. We're going to get could be wrong. We're going to get Nigerians texting me instantly after Nigerians texting me instantly after Nigerians texting me instantly after this. this. this. I actually I've read quite a lot of I actually I've read quite a lot of I actually I've read quite a lot of books from there's a Nigerian author books from there's a Nigerian author books from there's a Nigerian author that's really good. Um but you know when that's really good. Um but you know when that's really good. Um but you know when you read a book on text you don't get to you read a book on text you don't get to you read a book on text you don't get to hear how hear how hear how so that so for that language um they so that so for that language um they so that so for that language um they only brought in 32,000 pairs right so only brought in 32,000 pairs right so only brought in 32,000 pairs right so that and the oh the smallest language is that and the oh the smallest language is that and the oh the smallest language is Yoruba um yeah still a big also said Yoruba um yeah still a big also said Yoruba um yeah still a big also said wrong that one they only brought 16,000 wrong that one they only brought 16,000 wrong that one they only brought 16,000 pairs right so for the range for this pairs right so for the range for this pairs right so for the range for this model was between 16,000 pairs to model was between 16,000 pairs to model was between 16,000 pairs to 234 million pairs um and that you So 234 million pairs um and that you So 234 million pairs um and that you So total they had 1.6 billion pairs, right?

total they had 1.6 billion pairs, right? total they had 1.6 billion pairs, right? So you know there's way less So you know there's way less So you know there's way less representation of um you know the the representation of um you know the the representation of um you know the the the some of these languages here. Uh and the some of these languages here. Uh and the some of these languages here. Uh and these pairs came from multilingual news. these pairs came from multilingual news. these pairs came from multilingual news. If you're wondering like where you know If you're wondering like where you know If you're wondering like where you know where do you find these news? Do they where do you find these news? Do they where do you find these news? Do they find like news articles that have been find like news articles that have been find like news articles that have been translated into all these languages translated into all these languages translated into all these languages because that's fairly high quality because that's fairly high quality because that's fairly high quality information. Good point. Yeah. news information. Good point. Yeah. news information. Good point. Yeah. news articles makes more sense than just articles makes more sense than just articles makes more sense than just random social media because random random social media because random random social media because random social media would would would lean the social media would would would lean the social media would would would lean the vectors towards casual rather than vectors towards casual rather than vectors towards casual rather than formal speaking. Like it's not a it formal speaking. Like it's not a it formal speaking. Like it's not a it could be a conversational representation could be a conversational representation could be a conversational representation of how people talk the language but not of how people talk the language but not of how people talk the language but not necessarily how you would write a news a necessarily how you would write a news a necessarily how you would write a news a newspaper article or if you were going newspaper article or if you were going newspaper article or if you were going to run your PhD thesis through a to run your PhD thesis through a to run your PhD thesis through a co-pilot. Formality is a is a vector, co-pilot. Formality is a is a vector, co-pilot. Formality is a is a vector, isn't it? Uh yeah. Yeah. I I think isn't it? Uh yeah. Yeah. I I think isn't it? Uh yeah. Yeah. I I think formality is a vector. Yes. part of the formality is a vector. Yes. part of the formality is a vector. Yes. part of the dimensions these dimensions we say dimensions these dimensions we say dimensions these dimensions we say multi-dimensional space we can visualize multi-dimensional space we can visualize multi-dimensional space we can visualize two and three dimensions we struggle to two and three dimensions we struggle to two and three dimensions we struggle to do four and then we're talking about do four and then we're talking about do four and then we're talking about like low dimensions within the context like low dimensions within the context like low dimensions within the context of a vector embedding is like 50 and of a vector embedding is like 50 and of a vector embedding is like 50 and thousands of dimensions is typical yeah thousands of dimensions is typical yeah thousands of dimensions is typical yeah so the openi text embedding 3 large so the openi text embedding 3 large so the openi text embedding 3 large model is 372 dimensions that's the model is 372 dimensions that's the model is 372 dimensions that's the largest model that I've worked with and largest model that I've worked with and largest model that I've worked with and it's so large it actually doesn't fit in it's so large it actually doesn't fit in it's so large it actually doesn't fit in a lot of vector databases so like PG a lot of vector databases so like PG a lot of vector databases so like PG Vector recently had to make a you know Vector recently had to make a you know Vector recently had to make a you know make a change just to be able to support make a change just to be able to support make a change just to be able to support this model because they they didn't have this model because they they didn't have this model because they they didn't have a notion of being because it's just a a notion of being because it's just a a notion of being because it's just a list of floatingoint numbers right so list of floatingoint numbers right so list of floatingoint numbers right so they they didn't have the ability to

they they didn't have the ability to they they didn't have the ability to store a list of floatingoint numbers store a list of floatingoint numbers store a list of floatingoint numbers that was greater than 2,00 numbers until that was greater than 2,00 numbers until that was greater than 2,00 numbers until very recently right wow so then there's very recently right wow so then there's very recently right wow so then there's this thing that's called the curse of this thing that's called the curse of this thing that's called the curse of dimensionality where like we're we're dimensionality where like we're we're dimensionality where like we're we're coming up with relationships happy coming up with relationships happy coming up with relationships happy cheerful joyful kindness you know, and cheerful joyful kindness you know, and cheerful joyful kindness you know, and then start moving on all different then start moving on all different then start moving on all different directions around, you know, you're directions around, you know, you're directions around, you know, you're you're you're kind of limiting the you're you're kind of limiting the you're you're kind of limiting the thesaurus as you look for ways to thesaurus as you look for ways to thesaurus as you look for ways to describe words that are close together describe words that are close together describe words that are close together and figuring out what dimensional like a and figuring out what dimensional like a and figuring out what dimensional like a dimensionality could be. Make the word dimensionality could be. Make the word dimensionality could be. Make the word more formal. Make the, you know, it up a more formal. Make the, you know, it up a more formal. Make the, you know, it up a bit more likeing could bit more likeing could bit more likeing could be how to spell recently. Uh like all of be how to spell recently. Uh like all of be how to spell recently. Uh like all of these dimensions are and they're not these dimensions are and they're not these dimensions are and they're not necessarily named, right? They're just necessarily named, right? They're just necessarily named, right? They're just dimensions. They don't have like there's dimensions. They don't have like there's dimensions. They don't have like there's no like if you have king or queen no like if you have king or queen no like if you have king or queen there's no gendered dimension but there there's no gendered dimension but there there's no gendered dimension but there is a dimension that if you said give me is a dimension that if you said give me is a dimension that if you said give me a monarch make them very male give me a a monarch make them very male give me a a monarch make them very male give me a monarch make them very female you'll get monarch make them very female you'll get monarch make them very female you'll get king and queen but that doesn't king and queen but that doesn't king and queen but that doesn't necessarily imply that the vector has a necessarily imply that the vector has a necessarily imply that the vector has a name is that correct well that's name is that correct well that's name is that correct well that's interesting so with so word tovec was interesting so with so word tovec was interesting so with so word tovec was the embedding model that we used for a the embedding model that we used for a the embedding model that we used for a long time and saying that's the one you long time and saying that's the one you long time and saying that's the one you can train yourself locally fairly easily can train yourself locally fairly easily can train yourself locally fairly easily and that one is a very very strictly and that one is a very very strictly and that one is a very very strictly based off semantic representation based based off semantic representation based based off semantic representation based on like fill in the blank, right? Like on like fill in the blank, right? Like on like fill in the blank, right? Like if this would go in the blank and this if this would go in the blank and this if this would go in the blank and this could also go in the blank then uh you could also go in the blank then uh you could also go in the blank then uh you know then that uh you know that makes know then that uh you know that makes know then that uh you know that makes them more similar and that wordtovec them more similar and that wordtovec them more similar and that wordtovec model is famous for being able to do

model is famous for being able to do model is famous for being able to do like vector math. So if you have king like vector math. So if you have king like vector math. So if you have king and you like minus queen you like get and you like minus queen you like get and you like minus queen you like get back the vector for uh wow female or back the vector for uh wow female or back the vector for uh wow female or male or something like that right? So male or something like that right? So male or something like that right? So you could actually do this vector math you could actually do this vector math you could actually do this vector math and and people have had a lot of fun and and people have had a lot of fun and and people have had a lot of fun doing you know doing vector math to see doing you know doing vector math to see doing you know doing vector math to see what happens when you know what happens what happens when you know what happens what happens when you know what happens when you add these vectors and what and when you add these vectors and what and when you add these vectors and what and whatnot. But if you take the new whatnot. But if you take the new whatnot. But if you take the new embedding models, they are not so strict embedding models, they are not so strict embedding models, they are not so strict with being semantic exact semantic with being semantic exact semantic with being semantic exact semantic representations like that. Like for representations like that. Like for representations like that. Like for example, the first OpenAI model, one of example, the first OpenAI model, one of example, the first OpenAI model, one of the first ones was text embedding the first ones was text embedding the first ones was text embedding AD00002. And that one if I search for AD00002. And that one if I search for AD00002. And that one if I search for what's most similar to dog, then I get what's most similar to dog, then I get what's most similar to dog, then I get back God as the number two most similar back God as the number two most similar back God as the number two most similar thing. It goes dog, cat, and then God. thing. It goes dog, cat, and then God. thing. It goes dog, cat, and then God. And then farther down that like is like And then farther down that like is like And then farther down that like is like number six is drug. number six is drug. number six is drug. So, okay, So, okay, So, okay, it's not it's not it's not uh this is a good question as to why. uh this is a good question as to why. uh this is a good question as to why. Like, it just depends on how they Like, it just depends on how they Like, it just depends on how they trained that model. And I'm not privy to trained that model. And I'm not privy to trained that model. And I'm not privy to how they trained it, but I assume that how they trained it, but I assume that how they trained it, but I assume that with that particular model, they must with that particular model, they must with that particular model, they must have been training it in kind of a have been training it in kind of a have been training it in kind of a character by character way because you character by character way because you character by character way because you kind of see these spelling similarities, kind of see these spelling similarities, kind of see these spelling similarities, right? Like I assume that dog and God right? Like I assume that dog and God right? Like I assume that dog and God are similar because if you accidentally are similar because if you accidentally are similar because if you accidentally typed it in reverse, you would get sure typed it in reverse, you would get sure typed it in reverse, you would get sure God. drug and dog are similar because God. drug and dog are similar because God. drug and dog are similar because you know you make you miss make some you know you make you miss make some you know you make you miss make some mistakes and suddenly your dog becomes a mistakes and suddenly your dog becomes a mistakes and suddenly your dog becomes a drug you know something like that um so drug you know something like that um so drug you know something like that um so I think that model was trained with like I think that model was trained with like I think that model was trained with like in a way that it was looking at in a way that it was looking at in a way that it was looking at character similarity and like kind of character similarity and like kind of character similarity and like kind of spelling similarity

spelling similarity spelling similarity um but then the newer models I did the um but then the newer models I did the um but then the newer models I did the same experiment there and they basically same experiment there and they basically same experiment there and they basically fixed it so now you know dog is doesn't fixed it so now you know dog is doesn't fixed it so now you know dog is doesn't is not directly similar to god and drug is not directly similar to god and drug is not directly similar to god and drug so when you say fixed it So that's a so when you say fixed it So that's a so when you say fixed it So that's a bias of a kind because someone is then bias of a kind because someone is then bias of a kind because someone is then tweaking and waiting and putting their tweaking and waiting and putting their tweaking and waiting and putting their thumb on the scale and deciding like thumb on the scale and deciding like thumb on the scale and deciding like what do we want this to do it. So these what do we want this to do it. So these what do we want this to do it. So these become less technical problems and more become less technical problems and more become less technical problems and more for lack of a better word PM issues like for lack of a better word PM issues like for lack of a better word PM issues like these are like program manager figuring these are like program manager figuring these are like program manager figuring out the requirements for a model. Um out the requirements for a model. Um out the requirements for a model. Um which yeah I mean maybe dog should be which yeah I mean maybe dog should be which yeah I mean maybe dog should be similar to God right like if you wanted similar to God right like if you wanted similar to God right like if you wanted something that incorporated semantic something that incorporated semantic something that incorporated semantic similarity. And another another cool similarity. And another another cool similarity. And another another cool example is that um when you do do example is that um when you do do example is that um when you do do multilingual stuff, right, I get vastly multilingual stuff, right, I get vastly multilingual stuff, right, I get vastly different results based off of whether I different results based off of whether I different results based off of whether I use accented characters or what we call use accented characters or what we call use accented characters or what we call like, you know, like normalized like, you know, like normalized like, you know, like normalized characters, you know, like the the characters, you know, like the the characters, you know, like the the strict asy character set. And a lot of strict asy character set. And a lot of strict asy character set. And a lot of times it's actually more I get better times it's actually more I get better times it's actually more I get better results when I don't use accents. And I results when I don't use accents. And I results when I don't use accents. And I imagine what happened there is during imagine what happened there is during imagine what happened there is during the embedding training phase, they might the embedding training phase, they might the embedding training phase, they might have normalized all the accents, right?

have normalized all the accents, right? have normalized all the accents, right? We don't know. That's like another We don't know. That's like another We don't know. That's like another decision that somebody made which was decision that somebody made which was decision that somebody made which was how do we deal with the fact that like how do we deal with the fact that like how do we deal with the fact that like other languages have accents, right? And other languages have accents, right? And other languages have accents, right? And some people ignore them and some ignore some people ignore them and some ignore some people ignore them and some ignore them. Yeah. So like I usually like I'll them. Yeah. So like I usually like I'll them. Yeah. So like I usually like I'll always enter in like um El Leon or you always enter in like um El Leon or you always enter in like um El Leon or you know however you say Lion King in know however you say Lion King in know however you say Lion King in Spanish, right? And you can do an Spanish, right? And you can do an Spanish, right? And you can do an accent, you can not do an accent and accent, you can not do an accent and accent, you can not do an accent and I'll get better results if I don't use I'll get better results if I don't use I'll get better results if I don't use an accent. And so that's another thing an accent. And so that's another thing an accent. And so that's another thing like if you are working with non-English like if you are working with non-English like if you are working with non-English languages you you do want to like languages you you do want to like languages you you do want to like experiment and try like see how the experiment and try like see how the experiment and try like see how the model is responding to different ways of model is responding to different ways of model is responding to different ways of you know treating the characters in that you know treating the characters in that you know treating the characters in that language. When When I do rag, when I do retrieval augmented I do rag, when I do retrieval augmented I do rag, when I do retrieval augmented generation on a large amount of generation on a large amount of generation on a large amount of PDFs and I need a vector database, what PDFs and I need a vector database, what PDFs and I need a vector database, what ends up getting stored and what is the ends up getting stored and what is the ends up getting stored and what is the relationship between all the text that relationship between all the text that relationship between all the text that I'm stacking on top of all the other I'm stacking on top of all the other I'm stacking on top of all the other text, right? Because there's the corpus text, right? Because there's the corpus text, right? Because there's the corpus of the model that I picked and then of the model that I picked and then of the model that I picked and then there's my folder full of PDFs in there's my folder full of PDFs in there's my folder full of PDFs in whatever language they're in.

whatever language they're in. whatever language they're in. Well, so when you do rag, you send your Well, so when you do rag, you send your Well, so when you do rag, you send your documents through a data ingestion documents through a data ingestion documents through a data ingestion pipeline, which just means that we look pipeline, which just means that we look pipeline, which just means that we look at the document, we try to extract the at the document, we try to extract the at the document, we try to extract the text from it. Uh maybe even extract the text from it. Uh maybe even extract the text from it. Uh maybe even extract the images and then split the text up into images and then split the text up into images and then split the text up into chunks. And you know, the chunks, you chunks. And you know, the chunks, you chunks. And you know, the chunks, you know, maybe are a couple paragraphs. Um know, maybe are a couple paragraphs. Um know, maybe are a couple paragraphs. Um you know, there's there's some research, you know, there's there's some research, you know, there's there's some research, there's the best chunk size, whatever. there's the best chunk size, whatever. there's the best chunk size, whatever. Like so you split them into chunks that Like so you split them into chunks that Like so you split them into chunks that are a few paragraphs. And then for each are a few paragraphs. And then for each are a few paragraphs. And then for each of those chunks, you compute the of those chunks, you compute the of those chunks, you compute the embedding for them. And then and those embedding for them. And then and those embedding for them. And then and those then you store both the chunk and the then you store both the chunk and the then you store both the chunk and the embedding and you know like the the page embedding and you know like the the page embedding and you know like the the page number and the file name it came from so number and the file name it came from so number and the file name it came from so that then you could make a citation that then you could make a citation that then you could make a citation later. So then when you're using your later. So then when you're using your later. So then when you're using your actual rag application and you do uh you actual rag application and you do uh you actual rag application and you do uh you know you ask a question then we use that know you ask a question then we use that know you ask a question then we use that question in order to search the question in order to search the question in order to search the database. So basically we make a query database. So basically we make a query database. So basically we make a query based off that. Uh we make a vector based off that. Uh we make a vector based off that. Uh we make a vector based off that question that you asked based off that question that you asked based off that question that you asked and then we compare that vector to the and then we compare that vector to the and then we compare that vector to the stored vector and then uh hopefully we stored vector and then uh hopefully we stored vector and then uh hopefully we also do like a keyword search just also do like a keyword search just also do like a keyword search just because keyword search is also still because keyword search is also still because keyword search is also still helpful. People everyone's obsessed with helpful. People everyone's obsessed with helpful. People everyone's obsessed with vectors but we still want to do full vectors but we still want to do full vectors but we still want to do full text search. So in ideal situation we're text search. So in ideal situation we're text search. So in ideal situation we're doing both a vector search and a full doing both a vector search and a full doing both a vector search and a full text search and then merging them text search and then merging them text search and then merging them together. Yeah. Yeah. Yeah. Yes, it's be together. Yeah. Yeah. Yeah. Yes, it's be together. Yeah. Yeah. Yeah. Yes, it's be a hybrid search. So, I always recommend a hybrid search. So, I always recommend a hybrid search. So, I always recommend hybrid search because oh my gosh, you hybrid search because oh my gosh, you hybrid search because oh my gosh, you get crazy stuff if you only do vector get crazy stuff if you only do vector get crazy stuff if you only do vector search because what people often forget search because what people often forget search because what people often forget is that if you do a vector search, you

is that if you do a vector search, you is that if you do a vector search, you will always find a result even if it's will always find a result even if it's will always find a result even if it's really far away, right? Even if they're really far away, right? Even if they're really far away, right? Even if they're vectors are, you know, pointing in vectors are, you know, pointing in vectors are, you know, pointing in opposite directions, if that's the opposite directions, if that's the opposite directions, if that's the closest vector, then that's the result closest vector, then that's the result closest vector, then that's the result you're going to get. And uh so it's you're going to get. And uh so it's you're going to get. And uh so it's really important that you find ways to really important that you find ways to really important that you find ways to avoid the noisiness of vector search avoid the noisiness of vector search avoid the noisiness of vector search because it can be very very noisy. Uh because it can be very very noisy. Uh because it can be very very noisy. Uh but yeah, so then yeah, so we you know but yeah, so then yeah, so we you know but yeah, so then yeah, so we you know we compare the vectors for your question we compare the vectors for your question we compare the vectors for your question versus the vectors that are already versus the vectors that are already versus the vectors that are already stored for that chunk and then get back stored for that chunk and then get back stored for that chunk and then get back the most relevant ones. Uh so yeah, so the most relevant ones. Uh so yeah, so the most relevant ones. Uh so yeah, so we've been talking a lot about vectors we've been talking a lot about vectors we've been talking a lot about vectors for words, but these days the way we're for words, but these days the way we're for words, but these days the way we're using vectors is often on something much using vectors is often on something much using vectors is often on something much more complex than words, right? And so, more complex than words, right? And so, more complex than words, right? And so, you know, it's it's cool to see how what you know, it's it's cool to see how what you know, it's it's cool to see how what sort of results we get back when we sort of results we get back when we sort of results we get back when we search dog and we get back cat, but what search dog and we get back cat, but what search dog and we get back cat, but what actually matters is like, okay, we have actually matters is like, okay, we have actually matters is like, okay, we have this user question. How is that getting this user question. How is that getting this user question. How is that getting us back the best results in terms of our us back the best results in terms of our us back the best results in terms of our document chunks? Yeah. And the idea of document chunks? Yeah. And the idea of document chunks? Yeah. And the idea of like best results comes down to user like best results comes down to user like best results comes down to user intent and correctness and intent and correctness and intent and correctness and expressiveness. And there there's the uh expressiveness. And there there's the uh expressiveness. And there there's the uh like if you say give me the latest like if you say give me the latest like if you say give me the latest information on diabetes research then information on diabetes research then information on diabetes research then you've indirectly biased it towards you've indirectly biased it towards you've indirectly biased it towards recency and you might end up getting the recency and you might end up getting the recency and you might end up getting the wrong results because latest could be wrong results because latest could be wrong results because latest could be from the '7s or it could be from last from the '7s or it could be from last from the '7s or it could be from last week. But it depends on how it thinks week. But it depends on how it thinks week. But it depends on how it thinks about latest and then you don't know if about latest and then you don't know if about latest and then you don't know if the data was ingested with the date the data was ingested with the date the data was ingested with the date being really like like was the date the being really like like was the date the being really like like was the date the the date of the of the document or the the date of the of the document or the the date of the of the document or the date of the metadata around the document

date of the metadata around the document date of the metadata around the document or some text within the document. I or some text within the document. I or some text within the document. I found temporal data searching to be found temporal data searching to be found temporal data searching to be really challenging in in Rev. Yeah, I really challenging in in Rev. Yeah, I really challenging in in Rev. Yeah, I would actually not use vectors at all would actually not use vectors at all would actually not use vectors at all for that aspect for temporal searching for that aspect for temporal searching for that aspect for temporal searching because the thing is that okay so first because the thing is that okay so first because the thing is that okay so first of all like vectors for numbers are like of all like vectors for numbers are like of all like vectors for numbers are like vector similarity between two numbers vector similarity between two numbers vector similarity between two numbers you know doesn't necessarily think mean you know doesn't necessarily think mean you know doesn't necessarily think mean what we think it means right um so what we think it means right um so what we think it means right um so anything that involves numbers that's anything that involves numbers that's anything that involves numbers that's you know once again where I'm going to you know once again where I'm going to you know once again where I'm going to use full text search but that specific use full text search but that specific use full text search but that specific question you're looking for latest what question you're looking for latest what question you're looking for latest what I would do so I always put a um a I would do so I always put a um a I would do so I always put a um a pre-processing stage in a rag flow, pre-processing stage in a rag flow, pre-processing stage in a rag flow, right? So I get your query, I look at it right? So I get your query, I look at it right? So I get your query, I look at it and say like, oh, are there some filters and say like, oh, are there some filters and say like, oh, are there some filters I should pull out of this query to to I should pull out of this query to to I should pull out of this query to to apply to the data before I do a search apply to the data before I do a search apply to the data before I do a search of the content? Right? So you say latest of the content? Right? So you say latest of the content? Right? So you say latest like, okay, I can turn that into a like, okay, I can turn that into a like, okay, I can turn that into a filter that says uh filter the data to filter that says uh filter the data to filter that says uh filter the data to only get me back things that were sent only get me back things that were sent only get me back things that were sent in the last uh one year, right? And so in the last uh one year, right? And so in the last uh one year, right? And so then we filter that down first and then then we filter that down first and then then we filter that down first and then do the vector search, right? So that is do the vector search, right? So that is do the vector search, right? So that is going to get you better results than going to get you better results than going to get you better results than relying on vector similarity for latest relying on vector similarity for latest relying on vector similarity for latest because that's just way too fuzzy and I because that's just way too fuzzy and I because that's just way too fuzzy and I would not trust I would not trust vector would not trust I would not trust vector would not trust I would not trust vector search for it. As we get towards the end search for it. As we get towards the end search for it. As we get towards the end of the the show, let me ask you this of the the show, let me ask you this of the the show, let me ask you this hard question.

hard question. hard question. is the distance between the kind of is the distance between the kind of is the distance between the kind of conversation that we're having here, conversation that we're having here, conversation that we're having here, which is still pretty basic, still which is still pretty basic, still which is still pretty basic, still pretty remedial, and the regular folks pretty remedial, and the regular folks pretty remedial, and the regular folks that are just wanting to ask the LLM a that are just wanting to ask the LLM a that are just wanting to ask the LLM a question and get an answer. Like, you question and get an answer. Like, you question and get an answer. Like, you don't need to understand how the don't need to understand how the don't need to understand how the internal combustion engine works to have internal combustion engine works to have internal combustion engine works to have a successful drive to the shops, but a successful drive to the shops, but a successful drive to the shops, but it's nice. It changes your relationship it's nice. It changes your relationship it's nice. It changes your relationship with the car if you understand what's with the car if you understand what's with the car if you understand what's going on inside. Your relationship with going on inside. Your relationship with going on inside. Your relationship with any chatbot is going to be very any chatbot is going to be very any chatbot is going to be very different than mine because you know different than mine because you know different than mine because you know more what's going on inside. But then more what's going on inside. But then more what's going on inside. But then non-technical parent or non-technical non-technical parent or non-technical non-technical parent or non-technical child is going to have a very different child is going to have a very different child is going to have a very different relationship. How much should people relationship. How much should people relationship. How much should people know about this stuff to be consumers of know about this stuff to be consumers of know about this stuff to be consumers of this stuff? this stuff? this stuff? Yeah, that's a great question and I so Yeah, that's a great question and I so Yeah, that's a great question and I so one of the most important things to me one of the most important things to me one of the most important things to me is just understanding the difference is just understanding the difference is just understanding the difference between getting an answer from rag between getting an answer from rag between getting an answer from rag versus getting an answer from the versus getting an answer from the versus getting an answer from the weights, right? Because if you ask an weights, right? Because if you ask an weights, right? Because if you ask an LLM a question and it's doesn't have any LLM a question and it's doesn't have any LLM a question and it's doesn't have any additional context and it's just based additional context and it's just based additional context and it's just based off what it's weights, then you know off what it's weights, then you know off what it's weights, then you know you're only getting back what's you know you're only getting back what's you know you're only getting back what's you know what it was trained on, right? and as what it was trained on, right? and as what it was trained on, right? and as that's the you know most likely to yield that's the you know most likely to yield that's the you know most likely to yield inaccurate data um because it doesn't inaccurate data um because it doesn't inaccurate data um because it doesn't have access to latest information uh have access to latest information uh have access to latest information uh versus if you're using it with any sort versus if you're using it with any sort versus if you're using it with any sort of rag tools like if you go to chatb of rag tools like if you go to chatb of rag tools like if you go to chatb like chatb can now search the web right like chatb can now search the web right like chatb can now search the web right and it can search the web if it thinks

and it can search the web if it thinks and it can search the web if it thinks it needs to and it'll give you citations it needs to and it'll give you citations it needs to and it'll give you citations right or if you go to bing.com if you do right or if you go to bing.com if you do right or if you go to bing.com if you do a search right so if you see citations a search right so if you see citations a search right so if you see citations then basically it did rag and and that's then basically it did rag and and that's then basically it did rag and and that's that's the big thing I try to tell that's the big thing I try to tell that's the big thing I try to tell people is like if you see citations people is like if you see citations people is like if you see citations that is rag. That's good because then it that is rag. That's good because then it that is rag. That's good because then it means that the LLM looked at some means that the LLM looked at some means that the LLM looked at some context and now you can go off and check context and now you can go off and check context and now you can go off and check at that context yourself and and then at that context yourself and and then at that context yourself and and then you are empowered if you you know if you you are empowered if you you know if you you are empowered if you you know if you choose to go and dig in to the sources choose to go and dig in to the sources choose to go and dig in to the sources and see if they actually agree with it. and see if they actually agree with it. and see if they actually agree with it. Uh so I at a high level like that's that Uh so I at a high level like that's that Uh so I at a high level like that's that is one thing I want people to understand is one thing I want people to understand is one thing I want people to understand because we had this whole debate a week because we had this whole debate a week because we had this whole debate a week ago on um blue sky about you know do ago on um blue sky about you know do ago on um blue sky about you know do does do LLM search the web and I contend does do LLM search the web and I contend does do LLM search the web and I contend that LLMs do not search the web? LLMs that LLMs do not search the web? LLMs that LLMs do not search the web? LLMs are still just next word predictors are still just next word predictors are still just next word predictors right? However, an LLM can look at your right? However, an LLM can look at your right? However, an LLM can look at your user question, suggest what would be a user question, suggest what would be a user question, suggest what would be a good web query, and then our code, our good web query, and then our code, our good web query, and then our code, our application code can then say, "Okay, application code can then say, "Okay, application code can then say, "Okay, we're going to go search the web. We're we're going to go search the web. We're we're going to go search the web. We're going to get back results, and we're going to get back results, and we're going to get back results, and we're going to feed it to LM and say, hey, going to feed it to LM and say, hey, going to feed it to LM and say, hey, here was the user's query. Here's the here was the user's query. Here's the here was the user's query. Here's the results. Answer the question." Right?

results. Answer the question." Right? results. Answer the question." Right? And the LM is going to generate text And the LM is going to generate text And the LM is going to generate text based off of all that stuff, right? So based off of all that stuff, right? So based off of all that stuff, right? So the LLM is not searching the web, but a the LLM is not searching the web, but a the LLM is not searching the web, but a LM is helping us decide what we should LM is helping us decide what we should LM is helping us decide what we should search for and then synthesizing an search for and then synthesizing an search for and then synthesizing an answer based off those results. But we answer based off those results. But we answer based off those results. But we the application developers are building the application developers are building the application developers are building the tool that searches the web. You the tool that searches the web. You the tool that searches the web. You know, I want to call out a really know, I want to call out a really know, I want to call out a really interesting example where I think it was interesting example where I think it was interesting example where I think it was um Rinovich asked some question like is um Rinovich asked some question like is um Rinovich asked some question like is it 2025 to Google and Google has their it 2025 to Google and Google has their it 2025 to Google and Google has their AI answers thing and it was like no it's AI answers thing and it was like no it's AI answers thing and it was like no it's 2024 right so it was clearly not doing 2024 right so it was clearly not doing 2024 right so it was clearly not doing any tool calls it was just telling us any tool calls it was just telling us any tool calls it was just telling us the date that it was trained on but then the date that it was trained on but then the date that it was trained on but then and then I tried it with like Bing which and then I tried it with like Bing which and then I tried it with like Bing which uses co-pilot and that gave me the uses co-pilot and that gave me the uses co-pilot and that gave me the correct answer but we don't know if they correct answer but we don't know if they correct answer but we don't know if they searched and then figured it out and searched and then figured it out and searched and then figured it out and then like the tool will call was the then like the tool will call was the then like the tool will call was the web. But the AI that I have been web. But the AI that I have been web. But the AI that I have been personally most the AI search engine I'm personally most the AI search engine I'm personally most the AI search engine I'm putting that in quotes that I was most putting that in quotes that I was most putting that in quotes that I was most impressed with is duck.go go because impressed with is duck.go go because impressed with is duck.go go because they do the search and the summarization they do the search and the summarization they do the search and the summarization appears to be what appears on the page appears to be what appears on the page appears to be what appears on the page and like that that sounds silly but it's and like that that sounds silly but it's and like that that sounds silly but it's like Google's web search AI thing seems like Google's web search AI thing seems like Google's web search AI thing seems to be the most black boxy of them where to be the most black boxy of them where to be the most black boxy of them where it's like I did some magic Google stuff it's like I did some magic Google stuff it's like I did some magic Google stuff and here's the answer but it's clear and here's the answer but it's clear and here's the answer but it's clear that with duck.go go the results appear that with duck.go go the results appear that with duck.go go the results appear and then the AI thinks and then it tells and then the AI thinks and then it tells and then the AI thinks and then it tells me basically what's happening on the me basically what's happening on the me basically what's happening on the page below you. And what I liked about page below you. And what I liked about page below you. And what I liked about that from a user interface perspective

that from a user interface perspective that from a user interface perspective is it's like, oh, that's very is it's like, oh, that's very is it's like, oh, that's very intuitive. It's it's the TLDDR of the intuitive. It's it's the TLDDR of the intuitive. It's it's the TLDDR of the results as opposed to trying to glean my results as opposed to trying to glean my results as opposed to trying to glean my intent and figure out the answer without intent and figure out the answer without intent and figure out the answer without looking at the results. And that makes looking at the results. And that makes looking at the results. And that makes it feel very comfortable. Yeah, that's it feel very comfortable. Yeah, that's it feel very comfortable. Yeah, that's that's really nice. That's one of the that's really nice. That's one of the that's really nice. That's one of the general best practices for design of AI general best practices for design of AI general best practices for design of AI based systems is transparency. Microsoft based systems is transparency. Microsoft based systems is transparency. Microsoft actually has this really nice hack actually has this really nice hack actually has this really nice hack design toolkit is what it's called and design toolkit is what it's called and design toolkit is what it's called and it has all these design guidelines but it has all these design guidelines but it has all these design guidelines but more importantly it has all these more importantly it has all these more importantly it has all these examples where they've like examples where they've like examples where they've like screenshotted apps from all over from screenshotted apps from all over from screenshotted apps from all over from Microsoft from Google etc where they say Microsoft from Google etc where they say Microsoft from Google etc where they say like here's a nice example of this like here's a nice example of this like here's a nice example of this design principle and one of the big ones design principle and one of the big ones design principle and one of the big ones is transparency right where you can see is transparency right where you can see is transparency right where you can see like okay the LLM is has this amount of like okay the LLM is has this amount of like okay the LLM is has this amount of confidence in its answer the LLM based confidence in its answer the LLM based confidence in its answer the LLM based its answer on these parts of the text its answer on these parts of the text its answer on these parts of the text right so as much transparency as you can right so as much transparency as you can right so as much transparency as you can give people because then maybe we can give people because then maybe we can give people because then maybe we can start helping people to understand what start helping people to understand what start helping people to understand what the LM LM is doing and what it isn't the LM LM is doing and what it isn't the LM LM is doing and what it isn't doing so they can use it fast. Yeah, doing so they can use it fast. Yeah, doing so they can use it fast. Yeah, that is great. I like looking at that is great. I like looking at that is great. I like looking at examples of uh like UX. I feel like examples of uh like UX. I feel like examples of uh like UX. I feel like that's a really interesting that's a really interesting that's a really interesting conversation. I had Maggie Appleton on conversation. I had Maggie Appleton on conversation. I had Maggie Appleton on the show a while back talking about what the show a while back talking about what the show a while back talking about what the UI for AI is and it's it's bigger the UI for AI is and it's it's bigger the UI for AI is and it's it's bigger than just a text box. There's so many than just a text box. There's so many than just a text box. There's so many good examples of like this is good examples of like this is good examples of like this is transparent and thoughtful and clear and transparent and thoughtful and clear and transparent and thoughtful and clear and responsible and fair responsible and fair responsible and fair without hiding stuff. And I feel like I

without hiding stuff. And I feel like I without hiding stuff. And I feel like I want a little sparkle or a little square want a little sparkle or a little square want a little sparkle or a little square around the thing on the page that says around the thing on the page that says around the thing on the page that says AI generated. I want to know when, why, AI generated. I want to know when, why, AI generated. I want to know when, why, and how this was done and did it express and how this was done and did it express and how this was done and did it express my intent. my intent. my intent. uh and knowing how all these learning at uh and knowing how all these learning at uh and knowing how all these learning at least from you how these how these least from you how these how these least from you how these how these vectors work and the complexity of it vectors work and the complexity of it vectors work and the complexity of it has been super helpful. Thanks for has been super helpful. Thanks for has been super helpful. Thanks for spending time with me today. spending time with me today. spending time with me today. Sure. Yeah. And for the record, I don't Sure. Yeah. And for the record, I don't Sure. Yeah. And for the record, I don't think most people need to understand how think most people need to understand how think most people need to understand how vector like the lay people probably vector like the lay people probably vector like the lay people probably don't have to understand vectors, but don't have to understand vectors, but don't have to understand vectors, but they are just really fun to to dig into. they are just really fun to to dig into. they are just really fun to to dig into. If you like learning about this stuff, If you like learning about this stuff, If you like learning about this stuff, you can check out Pamela's blog at you can check out Pamela's blog at you can check out Pamela's blog at blog.pamelafox.org. She has a great blog.pamelafox.org. She has a great blog.pamelafox.org. She has a great article from May about visualizing article from May about visualizing article from May about visualizing vectors and a lot of other great blog vectors and a lot of other great blog vectors and a lot of other great blog posts going back many years. This has posts going back many years. This has posts going back many years. This has been another episode of Hansel Minutes been another episode of Hansel Minutes been another episode of Hansel Minutes and we'll see you again next week. and we'll see you again next week. and we'll see you again next week. [Music]