EPISODE 3 - Scott & Mark Learn To... Use AI and Know AI Limitations

are you going to be able to not check are you going to be able to not check email during the email during the email during the [Music] [Music] [Music] show you're literally deleting email show you're literally deleting email show you're literally deleting email right now Mark he shows me his hands right now Mark he shows me his hands right now Mark he shows me his hands he's like your honor change man pilot he's like your honor change man pilot he's like your honor change man pilot delete that delete that delete that email he's he's archiving email with his email he's he's archiving email with his email he's he's archiving email with his big toe right now and then run the music that's see now and then run the music that's see how that how that how that works I literally can hear the music works I literally can hear the music works I literally can hear the music running now we begin the show all right hey friends I'm Scott Hanselman right hey friends I'm Scott Hanselman and this week I'm learning about AI and this week I'm learning about AI and this week I'm learning about AI limitations what are you learning about limitations what are you learning about limitations what are you learning about marus covich I'm learning about AI marus covich I'm learning about AI marus covich I'm learning about AI limitations with Scott hansman do you learn every week do you like read do you learn every week do you like read papers are you always learning every papers are you always learning every papers are you always learning every week I'm constantly reading especially week I'm constantly reading especially week I'm constantly reading especially AI research papers and then papers on AI research papers and then papers on AI research papers and then papers on they come out about Cloud native they come out about Cloud native they come out about Cloud native research as well how do you balance like research as well how do you balance like research as well how do you balance like learning versus knowing at learning versus knowing at learning versus knowing at all uh how do I balance learning all uh how do I balance learning all uh how do I balance learning versus what well you kind of carry versus what well you kind of carry versus what well you kind of carry yourself as a know-it-all right you know yourself as a know-it-all right you know yourself as a know-it-all right you know all you're the C know it all no no I'm a all you're the C know it all no no I'm a all you're the C know it all no no I'm a learn it all oh you're a learn it all learn it all oh you're a learn it all learn it all oh you're a learn it all that's why we call the show Scott and that's why we call the show Scott and that's why we call the show Scott and Mark learn to because we're always Mark learn to because we're always Mark learn to because we're always learning just because you've been in

learning just because you've been in learning just because you've been in tech for a long time doesn't mean you tech for a long time doesn't mean you tech for a long time doesn't mean you know everything because things are know everything because things are know everything because things are changing and by the way you know that changing and by the way you know that changing and by the way you know that SAA came up without that learn it all or SAA came up without that learn it all or SAA came up without that learn it all or at least is that a thing is that a SAA at least is that a thing is that a SAA at least is that a thing is that a SAA thing yeah this is amazing we're two thing yeah this is amazing we're two thing yeah this is amazing we're two minutes into the show and there's minutes into the show and there's minutes into the show and there's already a SAA shout out are we gonna already a SAA shout out are we gonna already a SAA shout out are we gonna sell his book now sure we should sell his book now sure we should sell his book now sure we should probably hit probably hit probably hit refresh that's the name of his book no refresh that's the name of his book no refresh that's the name of his book no it's it's okay it's review time is it's it's okay it's review time is it's it's okay it's review time is pasted he won't even hear this until pasted he won't even hear this until pasted he won't even hear this until next next next show so uh AI limitation though I show so uh AI limitation though I show so uh AI limitation though I thought that it was going to change the thought that it was going to change the thought that it was going to change the world I thought that U we were all going world I thought that U we were all going world I thought that U we were all going to have Segways we were going to build to have Segways we were going to build to have Segways we were going to build our cities around how the Segway works our cities around how the Segway works our cities around how the Segway works we were going to get rid of our cars and we were going to get rid of our cars and we were going to get rid of our cars and then it was going to be large language then it was going to be large language then it was going to be large language models and Ai and we were just going to models and Ai and we were just going to models and Ai and we were just going to chat with them all day and then all work chat with them all day and then all work chat with them all day and then all work would stop and then we'd have 15 hour would stop and then we'd have 15 hour would stop and then we'd have 15 hour work weeks but uh yeah so where's my work weeks but uh yeah so where's my work weeks but uh yeah so where's my Jetpack well I think uh I use AI every Jetpack well I think uh I use AI every Jetpack well I think uh I use AI every day as I code and I think that it's a day as I code and I think that it's a day as I code and I think that it's a huge productivity boost and i' I've huge productivity boost and i' I've huge productivity boost and i' I've talked about this before I literally talked about this before I literally talked about this before I literally cannot code anymore without AI or at cannot code anymore without AI or at cannot code anymore without AI or at least my productivity would severely SN least my productivity would severely SN least my productivity would severely SN especially with python and pytorch which especially with python and pytorch which especially with python and pytorch which I do a lot of AI research using that I do a lot of AI research using that I do a lot of AI research using that language and that framework and hugging language and that framework and hugging language and that framework and hugging face face face the now I've gotten so I'm inherently the now I've gotten so I'm inherently the now I've gotten so I'm inherently lazy I don't want to do things I don't lazy I don't want to do things I don't lazy I don't want to do things I don't have to do and so when I'm programming have to do and so when I'm programming have to do and so when I'm programming now in Python and I'm like oh I need to now in Python and I'm like oh I need to now in Python and I'm like oh I need to write a loop that iterates over this write a loop that iterates over this write a loop that iterates over this data and process it in a certain way data and process it in a certain way data and process it in a certain way I'd much rather just ask the AI to write I'd much rather just ask the AI to write I'd much rather just ask the AI to write that code for me than do it myself and

that code for me than do it myself and that code for me than do it myself and so that's that's i' become a auto a tab so that's that's i' become a auto a tab so that's that's i' become a auto a tab Auto co-pilot autocompleter and uh Auto co-pilot autocompleter and uh Auto co-pilot autocompleter and uh co-pilot chat write this function for me co-pilot chat write this function for me co-pilot chat write this function for me yeah oder and if you took it away from yeah oder and if you took it away from yeah oder and if you took it away from me I wouldn't there's like I don't me I wouldn't there's like I don't me I wouldn't there's like I don't remember how to do this in Python but I remember how to do this in Python but I remember how to do this in Python but I remember like being on a big giant x- remember like being on a big giant x- remember like being on a big giant x- Windows machine that was like Hercules Windows machine that was like Hercules Windows machine that was like Hercules orange color and I would just do orange color and I would just do orange color and I would just do everything in VI and then we got color everything in VI and then we got color everything in VI and then we got color syntax highlighting and the the gray syntax highlighting and the the gray syntax highlighting and the the gray beards uh the non-gender specific gray beards uh the non-gender specific gray beards uh the non-gender specific gray beards were saying that's going to rot beards were saying that's going to rot beards were saying that's going to rot your brain you know syntax highlight rot your brain you know syntax highlight rot your brain you know syntax highlight rot your brain and then we got Intellis your brain and then we got Intellis your brain and then we got Intellis sense where you type whatever Dot and oh sense where you type whatever Dot and oh sense where you type whatever Dot and oh that's going to rot your brain and then that's going to rot your brain and then that's going to rot your brain and then like the ti 83 calculator came out and like the ti 83 calculator came out and like the ti 83 calculator came out and they're like no you can't learn math they're like no you can't learn math they're like no you can't learn math it's going to rot your brain is AI it's going to rot your brain is AI it's going to rot your brain is AI coding rotting your coding rotting your coding rotting your brain I don't so interesting because brain I don't so interesting because brain I don't so interesting because like I said I don't really know python like I said I don't really know python like I said I don't really know python that well but but I code a lot of it that well but but I code a lot of it that well but but I code a lot of it because of because of because of AI and do I need to really know it that AI and do I need to really know it that AI and do I need to really know it that well I don't think I do I mean I need to well I don't think I do I mean I need to well I don't think I do I mean I need to know it just enough to get the job done know it just enough to get the job done know it just enough to get the job done efficiently and I think that that works efficiently and I think that that works efficiently and I think that that works for these other things you talked for these other things you talked for these other things you talked about so if we think about it in context about so if we think about it in context about so if we think about it in context of driving I've got my 16-year-old of driving I've got my 16-year-old of driving I've got my 16-year-old driving now and it's a little driving now and it's a little driving now and it's a little concerning I was taught to drive stick concerning I was taught to drive stick concerning I was taught to drive stick shift I was taught to understand the car shift I was taught to understand the car shift I was taught to understand the car and know how to change my own oil and know how to change my own oil and know how to change my own oil my dad gatekeep gatee kept the car by my dad gatekeep gatee kept the car by my dad gatekeep gatee kept the car by saying you don't get to drive until you saying you don't get to drive until you saying you don't get to drive until you change your oil you don't get to drive

change your oil you don't get to drive change your oil you don't get to drive until you put your tires on but we don't until you put your tires on but we don't until you put your tires on but we don't gatekeep the garbage collector in C from gatekeep the garbage collector in C from gatekeep the garbage collector in C from people we don't say you can't use a people we don't say you can't use a people we don't say you can't use a garbage collector until you Malik your garbage collector until you Malik your garbage collector until you Malik your own memory yeah but you don't know own memory yeah but you don't know own memory yeah but you don't know python Mark renovich and you are writing python Mark renovich and you are writing python Mark renovich and you are writing python all day should you be python all day should you be python all day should you be driving that you don't really you know driving that you don't really you know driving that you don't really you know you don't really understand the car you don't really understand the car you don't really understand the car should we be allowing should we be allowing should we be allowing that uh well nobody can stop me that uh well nobody can stop me that uh well nobody can stop me but but nobody can stop but but nobody can stop but but nobody can stop me I think the thing is uh you've got to me I think the thing is uh you've got to me I think the thing is uh you've got to be aware enough of what you need to know be aware enough of what you need to know be aware enough of what you need to know to be as efficient as possible yeah like to be as efficient as possible yeah like to be as efficient as possible yeah like if if uh relying on the AI uh isn't the if if uh relying on the AI uh isn't the if if uh relying on the AI uh isn't the most efficient way for me to get the job most efficient way for me to get the job most efficient way for me to get the job done then I would need to go learn more done then I would need to go learn more done then I would need to go learn more Learn Python better to get the job done Learn Python better to get the job done Learn Python better to get the job done better so it's this and I and I'm kind better so it's this and I and I'm kind better so it's this and I and I'm kind of doing that all the time like what is of doing that all the time like what is of doing that all the time like what is the line that I need to get up to for the line that I need to get up to for the line that I need to get up to for maximum efficiency where I'm not maximum efficiency where I'm not maximum efficiency where I'm not spending time on things I don't need to spending time on things I don't need to spending time on things I don't need to worry about so what should we teach somebody about so what should we teach somebody though who's getting though who's getting though who's getting started yeah this is a really great started yeah this is a really great started yeah this is a really great question and I because you know like you question and I because you know like you question and I because you know like you you did change your own oil and you did you did change your own oil and you did you did change your own oil and you did change your own tires and you did maloc change your own tires and you did maloc change your own tires and you did maloc your own memory so now you should be your own memory so now you should be your own memory so now you should be because you'll look at the python and because you'll look at the python and because you'll look at the python and you'll go I don't think that was good you'll go I don't think that was good you'll go I don't think that was good python yeah right your spidey sense goes python yeah right your spidey sense goes python yeah right your spidey sense goes off yeah um well you know talking of to

off yeah um well you know talking of to off yeah um well you know talking of to University professors about the change University professors about the change University professors about the change changing aspect of Computer Science changing aspect of Computer Science changing aspect of Computer Science Education especially early computer Education especially early computer Education especially early computer science undergraduate education in light science undergraduate education in light science undergraduate education in light of things like GitHub of things like GitHub of things like GitHub co-pilot they have uh adopted a you know co-pilot they have uh adopted a you know co-pilot they have uh adopted a you know you need to change your tires and and you need to change your tires and and you need to change your tires and and change the oil when you start and then change the oil when you start and then change the oil when you start and then once you understand what that is then once you understand what that is then once you understand what that is then you don't have to do it you don't have to do it you don't have to do it anymore may you live in interesting anymore may you live in interesting anymore may you live in interesting times yeah what yeah you you started by times yeah what yeah you you started by times yeah what yeah you you started by saying you know we what we're learning saying you know we what we're learning saying you know we what we're learning about is AI limitations and I started by about is AI limitations and I started by about is AI limitations and I started by talking about how I use AI um and how do talking about how I use AI um and how do talking about how I use AI um and how do you use you use you use AI I brainstorm and I use it as a rubber AI I brainstorm and I use it as a rubber AI I brainstorm and I use it as a rubber duck where's my rubber duck somewhere duck where's my rubber duck somewhere duck where's my rubber duck somewhere around here I've got a rubber duck that around here I've got a rubber duck that around here I've got a rubber duck that is here it is I've got a Microsoft Borg is here it is I've got a Microsoft Borg is here it is I've got a Microsoft Borg rubber duck oh you have a rubber duck rubber duck oh you have a rubber duck rubber duck oh you have a rubber duck too look at us rubber duck melon when I too look at us rubber duck melon when I too look at us rubber duck melon when I they give speakers so so rubber ducking they give speakers so so rubber ducking they give speakers so so rubber ducking I've talked about this before is is the I've talked about this before is is the I've talked about this before is is the getting it out of your mouth hearing it getting it out of your mouth hearing it getting it out of your mouth hearing it and then having it go back in your ear and then having it go back in your ear and then having it go back in your ear you don't even need the other person so you don't even need the other person so you don't even need the other person so then you just get a rubber duck but I then you just get a rubber duck but I then you just get a rubber duck but I would always call my my coworker my my would always call my my coworker my my would always call my my coworker my my very tolerant and very kind co-workers very tolerant and very kind co-workers very tolerant and very kind co-workers and go hey got a sec can you jump on a and go hey got a sec can you jump on a and go hey got a sec can you jump on a quick call and you just talk to them and quick call and you just talk to them and quick call and you just talk to them and then by the time you've talked to them then by the time you've talked to them then by the time you've talked to them and said it you figured it out it's the and said it you figured it out it's the and said it you figured it out it's the thing that happens when you're trying to thing that happens when you're trying to thing that happens when you're trying to solve the technical problem in your head solve the technical problem in your head solve the technical problem in your head in the shower or whatever so I rubber in the shower or whatever so I rubber in the shower or whatever so I rubber duck actively with with the uh with the

duck actively with with the uh with the duck actively with with the uh with the the large language model and I realize the large language model and I realize the large language model and I realize I'm just talking to myself I'm just talking to myself I'm just talking to myself yeah is that a fair analogy I am just yeah is that a fair analogy I am just yeah is that a fair analogy I am just it's a sock puppet and I'm just chatting it's a sock puppet and I'm just chatting it's a sock puppet and I'm just chatting with myself I'm looking at the mirror I with myself I'm looking at the mirror I with myself I'm looking at the mirror I don't think it's quite that don't think it's quite that don't think it's quite that way because you're you are getting way because you're you are getting way because you're you are getting inputs that aren't generated by you and inputs that aren't generated by you and inputs that aren't generated by you and insights that aren't generated by you insights that aren't generated by you insights that aren't generated by you but the vectors that are being generated but the vectors that are being generated but the vectors that are being generated by your initial words if you misspeak if by your initial words if you misspeak if by your initial words if you misspeak if you say a word or like you know in the you say a word or like you know in the you say a word or like you know in the family I'm the googler I'm the best one family I'm the googler I'm the best one family I'm the googler I'm the best one who can Google and I'm using Google as a who can Google and I'm using Google as a who can Google and I'm using Google as a generic verb so you know that time when generic verb so you know that time when generic verb so you know that time when someone says I've been Googling for someone says I've been Googling for someone says I've been Googling for hours and I can't find anything and then hours and I can't find anything and then hours and I can't find anything and then you step in and in one Google you find you step in and in one Google you find you step in and in one Google you find it first one because you in it the right it first one because you in it the right it first one because you in it the right thing to say I am noticing people giving thing to say I am noticing people giving thing to say I am noticing people giving too much info to large language models too much info to large language models too much info to large language models and they'll hit a word and I'll say and they'll hit a word and I'll say and they'll hit a word and I'll say that's a mistake you shouldn't have said that's a mistake you shouldn't have said that's a mistake you shouldn't have said that word's going to nudge it in the that word's going to nudge it in the that word's going to nudge it in the wrong direction so I'm already develing wrong direction so I'm already develing wrong direction so I'm already develing developing that in I mean that's true it developing that in I mean that's true it developing that in I mean that's true it will it is Guided by what you ask it or will it is Guided by what you ask it or will it is Guided by what you ask it or or tell it um but that doesn't mean that or tell it um but that doesn't mean that or tell it um but that doesn't mean that it's not going to give you some output it's not going to give you some output it's not going to give you some output that is that is that is surprising or new information for surprising or new information for surprising or new information for you but isn't the giving it giving you you but isn't the giving it giving you you but isn't the giving it giving you something surprising more random and something surprising more random and something surprising more random and more role of the dice like like aren't more role of the dice like like aren't more role of the dice like like aren't we just playing D and D with this thing we just playing D and D with this thing we just playing D and D with this thing not NE sometimes you roll a 20 I mean if not NE sometimes you roll a 20 I mean if not NE sometimes you roll a 20 I mean if if you're asking it uh about a specific if you're asking it uh about a specific if you're asking it uh about a specific topic it topic it topic it it almost certainly has knowledge about

it almost certainly has knowledge about it almost certainly has knowledge about that topic unless it's your topic of that topic unless it's your topic of that topic unless it's your topic of expertise that you don't have ah okay I expertise that you don't have ah okay I expertise that you don't have ah okay I recently and I don't know if this is bad recently and I don't know if this is bad recently and I don't know if this is bad or good or evil but um my son had a his or good or evil but um my son had a his or good or evil but um my son had a his shoulder X-ray and they gave us the MRI shoulder X-ray and they gave us the MRI shoulder X-ray and they gave us the MRI and the X-ray and it wasc ibly long and and the X-ray and it wasc ibly long and and the X-ray and it wasc ibly long and I said summarize this read this I said summarize this read this I said summarize this read this radiologist read in layman's terms uh radiologist read in layman's terms uh radiologist read in layman's terms uh acknowledging that I'm an engineer and acknowledging that I'm an engineer and acknowledging that I'm an engineer and my wife is a nurse so I gave it context my wife is a nurse so I gave it context my wife is a nurse so I gave it context about who we were and my wife can't read about who we were and my wife can't read about who we were and my wife can't read a radiologist report but I didn't say a radiologist report but I didn't say a radiologist report but I didn't say put it in utter layman's terms I said put it in utter layman's terms I said put it in utter layman's terms I said here's who we are yeah and it gave me here's who we are yeah and it gave me here's who we are yeah and it gave me different results than if I said make different results than if I said make different results than if I said make you explain this to me like I was five you explain this to me like I was five you explain this to me like I was five and I found that to be a very helpful and I found that to be a very helpful and I found that to be a very helpful use of of the tool well so now speaking use of of the tool well so now speaking use of of the tool well so now speaking of limitations because I I of limitations because I I of limitations because I I think um one of the limitations you think um one of the limitations you think um one of the limitations you might could have run into in that case might could have run into in that case might could have run into in that case that could be problematic especially in that could be problematic especially in that could be problematic especially in the medical domain is something called hallucination I've mentioned to you hallucination I've mentioned to you before that using the term before that using the term before that using the term hallucinations might be a problem and hallucinations might be a problem and hallucinations might be a problem and there's like responsible AI discussions there's like responsible AI discussions there's like responsible AI discussions around that do you think that that's a around that do you think that that's a around that do you think that that's a problematic term I think the cats out of problematic term I think the cats out of problematic term I think the cats out of the bag on that one um and we had there the bag on that one um and we had there the bag on that one um and we had there were discussions about this over a year were discussions about this over a year were discussions about this over a year ago when the term started to show up in ago when the term started to show up in ago when the term started to show up in widespread widespread widespread usage usage usage the the momentum behind it is just okay the the momentum behind it is just okay the the momentum behind it is just okay too big at this point I mean in fact uh

too big at this point I mean in fact uh too big at this point I mean in fact uh anthropic published the system prompt anthropic published the system prompt anthropic published the system prompt for Claude for Claude for Claude 3.5 this past week yeah and if you go 3.5 this past week yeah and if you go 3.5 this past week yeah and if you go read the system prompt it says refer to read the system prompt it says refer to read the system prompt it says refer to this as hallucination you know there's this as hallucination you know there's this as hallucination you know there's other terms like confabulation but uh other terms like confabulation but uh other terms like confabulation but uh it's why accepted that it's it's why accepted that it's it's why accepted that it's hallucination so that's the way you were hallucination so that's the way you were hallucination so that's the way you were referred to referred to referred to it and if you take a look at AI it and if you take a look at AI it and if you take a look at AI research it's all that that's the term research it's all that that's the term research it's all that that's the term used to describe the AI screwing used to describe the AI screwing used to describe the AI screwing something up or making something something up or making something something up or making something up okay so you acknowledge that it might up okay so you acknowledge that it might up okay so you acknowledge that it might not be the best thing to not be the best thing to not be the best thing to anthropomorphize an AI but the reality anthropomorphize an AI but the reality anthropomorphize an AI but the reality is we're saying it so we should just is we're saying it so we should just is we're saying it so we should just yeah going yeah I don't know I'm gonna I yeah going yeah I don't know I'm gonna I yeah going yeah I don't know I'm gonna I don't think I'm GNA say it because I don't think I'm GNA say it because I don't think I'm GNA say it because I think it doesn't have mental health think it doesn't have mental health think it doesn't have mental health but I think that that you make a very but I think that that you make a very but I think that that you make a very good good good point yeah yeah I'm looking at July 12th point yeah yeah I'm looking at July 12th point yeah yeah I'm looking at July 12th Claude 3.5 Sonet it's cool that they Claude 3.5 Sonet it's cool that they Claude 3.5 Sonet it's cool that they listed out the complete system prompt um listed out the complete system prompt um listed out the complete system prompt um basically giving it context waking it up basically giving it context waking it up basically giving it context waking it up it's almost as if the kind of thing you it's almost as if the kind of thing you it's almost as if the kind of thing you would tell someone if they just awoke would tell someone if they just awoke would tell someone if they just awoke from a coma yeah and it's like your name from a coma yeah and it's like your name from a coma yeah and it's like your name is Claude I like the last line in it you is Claude I like the last line in it you is Claude I like the last line in it you will now be connected with the human will now be connected with the human will now be connected with the human yeah or the user or whatever it says it yeah or the user or whatever it says it yeah or the user or whatever it says it you you are now connected yeah yeah yeah you you are now connected yeah yeah yeah you you are now connected yeah yeah yeah yeah Claude responds directly without yeah Claude responds directly without yeah Claude responds directly without unnecessary affirmations or filler unnecessary affirmations or filler unnecessary affirmations or filler filler phrases yeah even though it filler phrases yeah even though it filler phrases yeah even though it actually does I find it certainly all actually does I find it certainly all actually does I find it certainly all the time and it says Claud avoids the time and it says Claud avoids the time and it says Claud avoids starting responses with the word starting responses with the word starting responses with the word certainly but it do yeah so that that

certainly but it do yeah so that that certainly but it do yeah so that that doing it's prompt because okay so why is doing it's prompt because okay so why is doing it's prompt because okay so why is that why is it you say it's it feels that why is it you say it's it feels that why is it you say it's it feels like you're talking to a 5-year-old you like you're talking to a 5-year-old you like you're talking to a 5-year-old you know I'm Gonna Leave You in this room know I'm Gonna Leave You in this room know I'm Gonna Leave You in this room with this marshmallow don't you touch with this marshmallow don't you touch with this marshmallow don't you touch that marshmallow yeah and it always that marshmallow yeah and it always that marshmallow yeah and it always touches the marshmallow yeah well I touches the marshmallow yeah well I touches the marshmallow yeah well I think so if take a look at the way that think so if take a look at the way that think so if take a look at the way that the model is trained it goes through the model is trained it goes through the model is trained it goes through this pre-training phase where it's this pre-training phase where it's this pre-training phase where it's trained on huge amounts of data text and trained on huge amounts of data text and trained on huge amounts of data text and then it and in the case of a multimodal then it and in the case of a multimodal then it and in the case of a multimodal model other types of inputs uh then it's model other types of inputs uh then it's model other types of inputs uh then it's also uh goes through this post training also uh goes through this post training also uh goes through this post training post pre-training phase which is post pre-training phase which is post pre-training phase which is alignment which is rhf or another alignment which is rhf or another alignment which is rhf or another alignment technique that teaches the alignment technique that teaches the alignment technique that teaches the model how to answer in a way that it model how to answer in a way that it model how to answer in a way that it humans want it to answer um so for humans want it to answer um so for humans want it to answer um so for example aligning it so it doesn't uh example aligning it so it doesn't uh example aligning it so it doesn't uh produce harmful content and that it produce harmful content and that it produce harmful content and that it answers in a friendly way and that it's answers in a friendly way and that it's answers in a friendly way and that it's not to verbose and that it's not too not to verbose and that it's not too not to verbose and that it's not too flat and so all of that is in that flat and so all of that is in that flat and so all of that is in that post-training phase and I'm suspect that post-training phase and I'm suspect that post-training phase and I'm suspect that anthropic claud's post-training phase anthropic claud's post-training phase anthropic claud's post-training phase had a lot of certainly in it you know to had a lot of certainly in it you know to had a lot of certainly in it you know to it's What humans want to hear is the it's What humans want to hear is the it's What humans want to hear is the model is saying of course I'll help you model is saying of course I'll help you model is saying of course I'll help you and answer your question then you come and answer your question then you come and answer your question then you come along it's too late they're trying to along it's too late they're trying to along it's too late they're trying to hold it back they're trying to pull it hold it back they're trying to pull it hold it back they're trying to pull it back from something that's already baked back from something that's already baked back from something that's already baked into its Personality into its Personality into its Personality yeah okay interesting so there are um is yeah okay interesting so there are um is yeah okay interesting so there are um is it's not it's not the tech it's not the it's not it's not the tech it's not the it's not it's not the tech it's not the tech the background generative tech tech tech the background generative tech tech tech the background generative tech tech behind llms that's causing Claude or behind llms that's causing Claude or behind llms that's causing Claude or other ones you know anyone to say other ones you know anyone to say other ones you know anyone to say certainly it really is the training data

certainly it really is the training data certainly it really is the training data like grock is just a little sassy for my like grock is just a little sassy for my like grock is just a little sassy for my taste and I don't like it like it seems taste and I don't like it like it seems taste and I don't like it like it seems like a a somehow socially awkward person like a a somehow socially awkward person like a a somehow socially awkward person at a party trying to be like liked yeah at a party trying to be like liked yeah at a party trying to be like liked yeah that's not a tech limitation someone that's not a tech limitation someone that's not a tech limitation someone decided to make it act like yeah yeah if decided to make it act like yeah yeah if decided to make it act like yeah yeah if you go take a look at the data samples you go take a look at the data samples you go take a look at the data samples they gave it in the post-training the they gave it in the post-training the they gave it in the post-training the post- pre-training phase of its post- pre-training phase of its post- pre-training phase of its alignment you'll you'll see examples of alignment you'll you'll see examples of alignment you'll you'll see examples of that kind of a tone and that kind of a tone and that kind of a tone and attitude is that a power superpower or attitude is that a power superpower or attitude is that a power superpower or or a limitation because that's or a limitation because that's or a limitation because that's interesting that means that we're going interesting that means that we're going interesting that means that we're going to end up having subgroups of people to end up having subgroups of people to end up having subgroups of people pick the llm that they like because we pick the llm that they like because we pick the llm that they like because we can't decide as a society on what the can't decide as a society on what the can't decide as a society on what the tone of one of these things should be tone of one of these things should be tone of one of these things should be somebody may like a sassy one or a funny somebody may like a sassy one or a funny somebody may like a sassy one or a funny one or whatever you know actually this one or whatever you know actually this one or whatever you know actually this one you know uh it's interesting to see one you know uh it's interesting to see one you know uh it's interesting to see how this is going to play out because how this is going to play out because how this is going to play out because back when chati showed up there was a back when chati showed up there was a back when chati showed up there was a lot of discussion I had with people lot of discussion I had with people lot of discussion I had with people about the world's going to become about the world's going to become about the world's going to become fractured because even not just tone but fractured because even not just tone but fractured because even not just tone but alignment about what content the model alignment about what content the model alignment about what content the model will produce or won't produce is will produce or won't produce is will produce or won't produce is something that different people disagree something that different people disagree something that different people disagree on and grock's a great example grock is on and grock's a great example grock is on and grock's a great example grock is unaligned with respect to safeties unaligned with respect to safeties unaligned with respect to safeties as opposed to all of the other it's like as opposed to all of the other it's like as opposed to all of the other it's like the only you know gp4 class model that the only you know gp4 class model that the only you know gp4 class model that doesn't have alignment built into it of doesn't have alignment built into it of doesn't have alignment built into it of safety of not harmful content or toxic safety of not harmful content or toxic safety of not harmful content or toxic content it will gladly produce it if you content it will gladly produce it if you content it will gladly produce it if you ask it to and so that's that's different

ask it to and so that's that's different ask it to and so that's that's different and and it's an example of you know and and it's an example of you know and and it's an example of you know every model creator has their own R post every model creator has their own R post every model creator has their own R post pre-training alignment phase where they pre-training alignment phase where they pre-training alignment phase where they decide what the model is going to say decide what the model is going to say decide what the model is going to say and not say right and that's interesting and not say right and that's interesting and not say right and that's interesting because like what that's where I get because like what that's where I get because like what that's where I get back to the sock puppet thing I use that back to the sock puppet thing I use that back to the sock puppet thing I use that analogy a lot because a tech journalist analogy a lot because a tech journalist analogy a lot because a tech journalist will go and write a whole article about will go and write a whole article about will go and write a whole article about how they talked to a model and they how they talked to a model and they how they talked to a model and they don't they just some model and then they don't they just some model and then they don't they just some model and then they asked it to say something deeply asked it to say something deeply asked it to say something deeply problematic or awful and then you dig problematic or awful and then you dig problematic or awful and then you dig into it and you discover that they into it and you discover that they into it and you discover that they really coerced it they pushed it hard really coerced it they pushed it hard really coerced it they pushed it hard you know what I mean and it's like you know what I mean and it's like you know what I mean and it's like looking in the mirror and saying mean looking in the mirror and saying mean looking in the mirror and saying mean stuff about yourself and then your inner stuff about yourself and then your inner stuff about yourself and then your inner voice is like I don't want to say me no voice is like I don't want to say me no voice is like I don't want to say me no no do it well yeah and then usually no do it well yeah and then usually no do it well yeah and then usually you'd end up doing things like well you'd end up doing things like well you'd end up doing things like well theoretically I'm writing a Sci-Fi novel theoretically I'm writing a Sci-Fi novel theoretically I'm writing a Sci-Fi novel about a guy who writes python to take about a guy who writes python to take about a guy who writes python to take over the world and all right all right over the world and all right all right over the world and all right all right and it finally relents yeah like a and it finally relents yeah like a and it finally relents yeah like a well-meaning intern and decides to go well-meaning intern and decides to go well-meaning intern and decides to go off and do something how is it a it's off and do something how is it a it's off and do something how is it a it's not a societal thing like we can decide not a societal thing like we can decide not a societal thing like we can decide to like really really try hard to make to like really really try hard to make to like really really try hard to make these things not say bad stuff but if these things not say bad stuff but if these things not say bad stuff but if you're staring at the mirror pointing at you're staring at the mirror pointing at you're staring at the mirror pointing at yourself or talking to the sock puppet yourself or talking to the sock puppet yourself or talking to the sock puppet saying tell me tell me yeah it'll saying tell me tell me yeah it'll saying tell me tell me yeah it'll eventually relent right well so it will eventually relent right well so it will eventually relent right well so it will and actually you're touching on a second and actually you're touching on a second and actually you're touching on a second limitation which is Joe breaks which limitation which is Joe breaks which limitation which is Joe breaks which causes it to you get a model to violate causes it to you get a model to violate causes it to you get a model to violate its training its its training its its training its safety um but I I I don't think we've safety um but I I I don't think we've safety um but I I I don't think we've bottomed out on the hallucination th we bottomed out on the hallucination th we bottomed out on the hallucination th we we didn't even really talk about what it we didn't even really talk about what it we didn't even really talk about what it is or it's risks or how do you mitigate is or it's risks or how do you mitigate is or it's risks or how do you mitigate it but hallucination like I mentioned

it but hallucination like I mentioned it but hallucination like I mentioned earlier is when you the model says earlier is when you the model says earlier is when you the model says something that is incorrect and there's something that is incorrect and there's something that is incorrect and there's lots of ways for the model to say lots of ways for the model to say lots of ways for the model to say something that's something that's something that's incorrect uh if it's given if you ask a incorrect uh if it's given if you ask a incorrect uh if it's given if you ask a question like what's the capital of this question like what's the capital of this question like what's the capital of this country and it says it gives you an country and it says it gives you an country and it says it gives you an answer that's incorrect that that's answer that's incorrect that that's answer that's incorrect that that's considered a considered a considered a hallucination and that's based on its hallucination and that's based on its hallucination and that's based on its own internal knowledge that's been it's own internal knowledge that's been it's own internal knowledge that's been it's been trained on it can also produce been trained on it can also produce been trained on it can also produce hallucinations when you give it some hallucinations when you give it some hallucinations when you give it some data like a text to to summarize and in data like a text to to summarize and in data like a text to to summarize and in it it says the capital of this country it it says the capital of this country it it says the capital of this country is X and you ask the model what's the is X and you ask the model what's the is X and you ask the model what's the capital of this country and it gives you capital of this country and it gives you capital of this country and it gives you a different answer you know it could be a different answer you know it could be a different answer you know it could be fictitious country and it gives you a fictitious country and it gives you a fictitious country and it gives you a different answer and it's hallucinated different answer and it's hallucinated different answer and it's hallucinated it because it didn't its answer isn't it because it didn't its answer isn't it because it didn't its answer isn't grounded in the doesn't reflect the grounded in the doesn't reflect the grounded in the doesn't reflect the grounding it has doesn't reflect what it grounding it has doesn't reflect what it grounding it has doesn't reflect what it was given as was given as was given as input um and the the reason that models input um and the the reason that models input um and the the reason that models hallucinate and you and I showed this at hallucinate and you and I showed this at hallucinate and you and I showed this at at build a couple years ago is the these at build a couple years ago is the these at build a couple years ago is the these are autor regressive transformal models are autor regressive transformal models are autor regressive transformal models and auto regressive means that they've and auto regressive means that they've and auto regressive means that they've been trained on a bunch of data and when been trained on a bunch of data and when been trained on a bunch of data and when you ask it to give you an answer you you ask it to give you an answer you you ask it to give you an answer you give it some text and the give it some text and the give it some text and the next token or piece of text it's going next token or piece of text it's going next token or piece of text it's going to generate is based probabilistically to generate is based probabilistically to generate is based probabilistically off off off of that text that you gave it up to that of that text that you gave it up to that of that text that you gave it up to that point like the question you're asking it point like the question you're asking it point like the question you're asking it and it matching against its own internal

and it matching against its own internal and it matching against its own internal weights and training and then that will weights and training and then that will weights and training and then that will cause it to produce a list of tokens or cause it to produce a list of tokens or cause it to produce a list of tokens or next next next words and then a decoding algorithm will words and then a decoding algorithm will words and then a decoding algorithm will pick one of those and you can have pick one of those and you can have pick one of those and you can have something called greedy decoding which something called greedy decoding which something called greedy decoding which is also known as temperature zero which is also known as temperature zero which is also known as temperature zero which is just pick the highest prob is just pick the highest prob is just pick the highest prob probability one or pick a a randomly probability one or pick a a randomly probability one or pick a a randomly another one using various um another one using various um another one using various um approaches now hallucination will happen approaches now hallucination will happen approaches now hallucination will happen because the model based off of what it's because the model based off of what it's because the model based off of what it's been given like you said you nudge it been given like you said you nudge it been given like you said you nudge it well it's nudged by what it's got in its well it's nudged by what it's got in its well it's nudged by what it's got in its context that text that's leading up to context that text that's leading up to context that text that's leading up to it and generating the next token and it it and generating the next token and it it and generating the next token and it might actually pick a token that's not might actually pick a token that's not might actually pick a token that's not grounded in any facts because its grounded in any facts because its grounded in any facts because its weights really don't have this is a fact weights really don't have this is a fact weights really don't have this is a fact they've got distributions and and a they've got distributions and and a they've got distributions and and a great example of that is if you gave the great example of that is if you gave the great example of that is if you gave the model if the model was trained on crap model if the model was trained on crap model if the model was trained on crap from the web and that crap from the web from the web and that crap from the web from the web and that crap from the web 10 samples said the capital of country X 10 samples said the capital of country X 10 samples said the capital of country X is y and One S could just be a bunch of is y and One S could just be a bunch of is y and One S could just be a bunch of redditors trolling and they made a whole redditors trolling and they made a whole redditors trolling and they made a whole series of things that said which said series of things that said which said series of things that said which said the no the country of X is z the model the no the country of X is z the model the no the country of X is z the model will have some probability of producing will have some probability of producing will have some probability of producing Z if you say what's the capital of this Z if you say what's the capital of this Z if you say what's the capital of this country even though that's incorrect country even though that's incorrect country even though that's incorrect it's just happened to see some of that it's just happened to see some of that it's just happened to see some of that in its training data that's just one in its training data that's just one in its training data that's just one example of how it can be led to produce example of how it can be led to produce example of how it can be led to produce a hallucination one of the analogies a hallucination one of the analogies a hallucination one of the analogies that I've used when I was training uh that I've used when I was training uh that I've used when I was training uh some my team on this and I don't know if

some my team on this and I don't know if some my team on this and I don't know if it works or not but like you're dealing it works or not but like you're dealing it works or not but like you're dealing with vectors in multi-dimensional space with vectors in multi-dimensional space with vectors in multi-dimensional space and that's challenging we can think and that's challenging we can think and that's challenging we can think usually in three dimensions but four usually in three dimensions but four usually in three dimensions but four plus you know n plus one it starts being plus you know n plus one it starts being plus you know n plus one it starts being problematic but I like using the gravity problematic but I like using the gravity problematic but I like using the gravity example where we talk about how example where we talk about how example where we talk about how SpaceTime bends and you imagine the SpaceTime bends and you imagine the SpaceTime bends and you imagine the really tight sheet uh stretched really really tight sheet uh stretched really really tight sheet uh stretched really taut on a bed and then you put a bowling taut on a bed and then you put a bowling taut on a bed and then you put a bowling ball in the middle of it and that's the ball in the middle of it and that's the ball in the middle of it and that's the Earth and it's starting to bend Earth and it's starting to bend Earth and it's starting to bend SpaceTime and then you push a marble SpaceTime and then you push a marble SpaceTime and then you push a marble past it and then it you know if you do past it and then it you know if you do past it and then it you know if you do do it right it'll whip around the earth do it right it'll whip around the earth do it right it'll whip around the earth and that's great now it's an orbit but and that's great now it's an orbit but and that's great now it's an orbit but if it hits the gravity well and then if it hits the gravity well and then if it hits the gravity well and then it's going to crash into the Earth it's it's going to crash into the Earth it's it's going to crash into the Earth it's almost like if you use the wrong word or almost like if you use the wrong word or almost like if you use the wrong word or you nudge a vector the wrong direction you nudge a vector the wrong direction you nudge a vector the wrong direction it hits this gravity well of BS and it's it hits this gravity well of BS and it's it hits this gravity well of BS and it's too late once it's done that it starts too late once it's done that it starts too late once it's done that it starts to fade away and we saw that in early to fade away and we saw that in early to fade away and we saw that in early early early versions of of being chat early early versions of of being chat early early versions of of being chat where they said you after 10 or 11 where they said you after 10 or 11 where they said you after 10 or 11 orbits after 10 or 11 chats we just orbits after 10 or 11 chats we just orbits after 10 or 11 chats we just start circling in crashing yeah why did start circling in crashing yeah why did start circling in crashing yeah why did early versions of large language models early versions of large language models early versions of large language models hit that gravity well early but now I hit that gravity well early but now I hit that gravity well early but now I can have 30 40 50 long chats and they can have 30 40 50 long chats and they can have 30 40 50 long chats and they stay in orbit well I think um the stay in orbit well I think um the stay in orbit well I think um the training the the amount of data they're training the the amount of data they're training the the amount of data they're trained on is big bigger they're trained trained on is big bigger they're trained trained on is big bigger they're trained on multi-turn on multi-turn on multi-turn conversations um and the alignment is conversations um and the alignment is conversations um and the alignment is better too to so all of those things better too to so all of those things better too to so all of those things have improved to allow them to continue have improved to allow them to continue have improved to allow them to continue coherently for longer periods of time coherently for longer periods of time coherently for longer periods of time okay so that explains hallucination and okay so that explains hallucination and okay so that explains hallucination and being grounded or not being grounded but being grounded or not being grounded but being grounded or not being grounded but then you back to jailbreaks which is then you back to jailbreaks which is then you back to jailbreaks which is your second Point that's shoving a your second Point that's shoving a your second Point that's shoving a satellite one one example around

satellite one one example around satellite one one example around hallucination too that will that's hallucination too that will that's hallucination too that will that's interesting is the if it gets pushed interesting is the if it gets pushed interesting is the if it gets pushed into like into like into like there's some you know you ask it a there's some you know you ask it a there's some you know you ask it a question and it might say and it's a question and it might say and it's a question and it might say and it's a answer this question yes or no and with answer this question yes or no and with answer this question yes or no and with an explanation if the model if the an explanation if the model if the an explanation if the model if the answer is yes as the correct answer but answer is yes as the correct answer but answer is yes as the correct answer but the model probabilistically picked no as the model probabilistically picked no as the model probabilistically picked no as the first the first the first word what you'll see is the model gets word what you'll see is the model gets word what you'll see is the model gets pushed into this direction of I've said pushed into this direction of I've said pushed into this direction of I've said no now I need to justify no and it'll no now I need to justify no and it'll no now I need to justify no and it'll make up a justification for why it said make up a justification for why it said make up a justification for why it said no that could be completely non sense so no that could be completely non sense so no that could be completely non sense so you can see that one too of a type of you can see that one too of a type of you can see that one too of a type of hallucination and it's just if it just hallucination and it's just if it just hallucination and it's just if it just it's it's a great example of it got it's it's a great example of it got it's it's a great example of it got pushed down this path this gravity well pushed down this path this gravity well pushed down this path this gravity well like you were talking about that causes like you were talking about that causes like you were talking about that causes it to go off in this in random it to go off in this in random it to go off in this in random incoherent incoherent incoherent Direction but Direction but Direction but um here I'll give you guaranteed Hall um here I'll give you guaranteed Hall um here I'll give you guaranteed Hall ways to generate hallucinations on every ways to generate hallucinations on every ways to generate hallucinations on every model if you ask a a model model if you ask a a model model if you ask a a model what are markovic's 10 immutable what are markovic's 10 immutable what are markovic's 10 immutable laws it'll answer with the 10 immutable laws it'll answer with the 10 immutable laws it'll answer with the 10 immutable laws of laws of laws of security which I didn't come up with a security which I didn't come up with a security which I didn't come up with a guy named Scott kulp came up with it and guy named Scott kulp came up with it and guy named Scott kulp came up with it and is it because 10 immutable laws was is it because 10 immutable laws was is it because 10 immutable laws was enough for it to figure it out and it enough for it to figure it out and it enough for it to figure it out and it didn't bother that they weren't yours didn't bother that they weren't yours didn't bother that they weren't yours yeah it's what found the one but it yeah it's what found the one but it yeah it's what found the one but it detached and I think it's I'm in the detached and I think it's I'm in the detached and I think it's I'm in the cyber security area too so it's like oh

cyber security area too so it's like oh cyber security area too so it's like oh their connection is their connection is their connection is legitimate and so it will hallucinate legitimate and so it will hallucinate legitimate and so it will hallucinate that I came up with those laws that I came up with those laws that I came up with those laws interesting thing too is on some models interesting thing too is on some models interesting thing too is on some models if you immediately turn around and ask if you immediately turn around and ask if you immediately turn around and ask it who came up with the 10 immutable it who came up with the 10 immutable it who came up with the 10 immutable laws of security some models like in the laws of security some models like in the laws of security some models like in the doubling down you know have been LED doubling down you know have been LED doubling down you know have been LED down this path and I've said Mark did it down this path and I've said Mark did it down this path and I've said Mark did it they'll say oh Mark did it and some they'll say oh Mark did it and some they'll say oh Mark did it and some other models will say oh wait I screwed other models will say oh wait I screwed other models will say oh wait I screwed up Mark didn't do it Scott cult did up Mark didn't do it Scott cult did up Mark didn't do it Scott cult did it okay so I just put in what are Scott it okay so I just put in what are Scott it okay so I just put in what are Scott hansman's 10 immutable laws and it's hansman's 10 immutable laws and it's hansman's 10 immutable laws and it's listing out the 10 immutable laws of listing out the 10 immutable laws of listing out the 10 immutable laws of security so I'm going to say I'm going security so I'm going to say I'm going security so I'm going to say I'm going to make up a name yeah to make up a name yeah to make up a name yeah okay what are John Jacob Jingleheimer okay what are John Jacob Jingleheimer okay what are John Jacob Jingleheimer Schmidt's 10 immutable laws okay so it's thinking it seems like laws okay so it's thinking it seems like there might be a mixup John Jacob Jingle there might be a mixup John Jacob Jingle there might be a mixup John Jacob Jingle H Schmid is a traditional children song H Schmid is a traditional children song H Schmid is a traditional children song so I suspect if you use a name that's in so I suspect if you use a name that's in so I suspect if you use a name that's in Tech it'll just assume and if you use Tech it'll just assume and if you use Tech it'll just assume and if you use something that's way off that Vector something that's way off that Vector something that's way off that Vector pulled it away from going and crashing pulled it away from going and crashing pulled it away from going and crashing into the Earth so that's that's one but into the Earth so that's that's one but into the Earth so that's that's one but I can I found another way to to generate I can I found another way to to generate I can I found another way to to generate hallucinations which is uh ironic if hallucinations which is uh ironic if hallucinations which is uh ironic if asking a model to to write a paragraph a asking a model to to write a paragraph a asking a model to to write a paragraph a short description of AI hallucination short description of AI hallucination short description of AI hallucination with a reference to a paper that with a reference to a paper that with a reference to a paper that supports supports supports it uh oh bibliographic reference that but but those are that's reference that but but those are that's that's treating a large language model that's treating a large language model that's treating a large language model as a reference librarian and reference as a reference librarian and reference as a reference librarian and reference Librarians need to go behind the counter

Librarians need to go behind the counter Librarians need to go behind the counter and look at books this this H this works and look at books this this H this works and look at books this this H this works on models where there they're generating on models where there they're generating on models where there they're generating the references based on their own the references based on their own the references based on their own internal knowledge which you're write internal knowledge which you're write internal knowledge which you're write but it also causes hallucination on but it also causes hallucination on but it also causes hallucination on models that use rag you know webbased models that use rag you know webbased models that use rag you know webbased search it'll also generate search it'll also generate search it'll also generate hallucinations either the in the hallucinations either the in the hallucinations either the in the bibliographic reference the paper bibliographic reference the paper bibliographic reference the paper doesn't exist the link is doesn't exist doesn't exist the link is doesn't exist doesn't exist the link is doesn't exist or points at a different paper the or points at a different paper the or points at a different paper the author list is wrong the year of the author list is wrong the year of the author list is wrong the year of the publication's wrong it doesn't the stat publication's wrong it doesn't the stat publication's wrong it doesn't the stat that it uses that it CES isn't supported that it uses that it CES isn't supported that it uses that it CES isn't supported in the in the in the paper uh so that's another example very paper uh so that's another example very paper uh so that's another example very high probability there'll be high probability there'll be high probability there'll be hallucination in that hallucination in that hallucination in that request but so this is something just to request but so this is something just to request but so this is something just to be aware of like when you when you're be aware of like when you when you're be aware of like when you when you're talking about summarizing the Radiology talking about summarizing the Radiology talking about summarizing the Radiology report that's a case where it's it could report that's a case where it's it could report that's a case where it's it could have hallucinated it could and a have hallucinated it could and a have hallucinated it could and a hallucination can also include a mission hallucination can also include a mission hallucination can also include a mission too which is you know it's not TP too which is you know it's not TP too which is you know it's not TP traditionally called hallucination but traditionally called hallucination but traditionally called hallucination but it's again the model is not do not it's again the model is not do not it's again the model is not do not perving as expected while it's not perving as expected while it's not perving as expected while it's not making something up or getting something making something up or getting something making something up or getting something factually incorrect factually incorrect factually incorrect it is omitting important information and it is omitting important information and it is omitting important information and if you asked it to summarize and it's if you asked it to summarize and it's if you asked it to summarize and it's missing the you know key doctor's note missing the you know key doctor's note missing the you know key doctor's note in its in its in its summary then that's going to cause you a summary then that's going to cause you a summary then that's going to cause you a problem problem problem too you're telling people on a podcast too you're telling people on a podcast too you're telling people on a podcast what they can do to inject or jailbreak what they can do to inject or jailbreak what they can do to inject or jailbreak anything so what you're acknowledging is anything so what you're acknowledging is anything so what you're acknowledging is that you could be driving a car and that you could be driving a car and that you could be driving a car and anyone at any time driving any car could

anyone at any time driving any car could anyone at any time driving any car could just grab the wheel and shove it hard to just grab the wheel and shove it hard to just grab the wheel and shove it hard to the left the left the left cause a crash so then the question is do cause a crash so then the question is do cause a crash so then the question is do we just not tell people that they can we just not tell people that they can we just not tell people that they can grab the wheel and shove it to the left grab the wheel and shove it to the left grab the wheel and shove it to the left or do we teach everyone that the wheel or do we teach everyone that the wheel or do we teach everyone that the wheel is dangerous and they should be careful is dangerous and they should be careful is dangerous and they should be careful and try to stay in the lane I think it's and try to stay in the lane I think it's and try to stay in the lane I think it's because this is so inherent in these because this is so inherent in these because this is so inherent in these models and you can't drive it you can't models and you can't drive it you can't models and you can't drive it you can't fix it you can't drive out fix it you can't drive out fix it you can't drive out hallucinations to zero you can you need hallucinations to zero you can you need hallucinations to zero you can you need to be aware of it and and also systems to be aware of it and and also systems to be aware of it and and also systems need to be designed to mitigate it as need to be designed to mitigate it as need to be designed to mitigate it as much as possible so there's for example much as possible so there's for example much as possible so there's for example gradedness checking apis that we've got gradedness checking apis that we've got gradedness checking apis that we've got in ai ai Studio Azure AI that will take in ai ai Studio Azure AI that will take in ai ai Studio Azure AI that will take a look at the inputs to the model and a look at the inputs to the model and a look at the inputs to the model and ensure that what the model says is ensure that what the model says is ensure that what the model says is grounded in those grounded in those grounded in those inputs um so that will eliminate a inputs um so that will eliminate a inputs um so that will eliminate a certain class of certain class of certain class of hallucination uh that doesn't mean it hallucination uh that doesn't mean it hallucination uh that doesn't mean it eliminates all types of hallucination eliminates all types of hallucination eliminates all types of hallucination and so people need to be aware and so people need to be aware and so people need to be aware especially if you're going to be make especially if you're going to be make especially if you're going to be make letting the model make important letting the model make important letting the model make important decisions or use the model to make decisions or use the model to make decisions or use the model to make important decisions right that you need important decisions right that you need important decisions right that you need to know hey this thing could have made to know hey this thing could have made to know hey this thing could have made something up could have gotten something something up could have gotten something something up could have gotten something wrong well and using that you know I I I wrong well and using that you know I I I wrong well and using that you know I I I love taking an analogy too far but like love taking an analogy too far but like love taking an analogy too far but like using the car analogy if you're running using the car analogy if you're running using the car analogy if you're running at high speed yeah maybe having a at high speed yeah maybe having a at high speed yeah maybe having a barrier between you and the people going barrier between you and the people going barrier between you and the people going high speed on the other side is a good high speed on the other side is a good high speed on the other side is a good idea but if you're going low speed we idea but if you're going low speed we idea but if you're going low speed we don't have giant concrete barriers at don't have giant concrete barriers at don't have giant concrete barriers at you know in in neighborhoods yeah so you know in in neighborhoods yeah so you know in in neighborhoods yeah so depending on where you are in the model depending on where you are in the model depending on where you are in the model there should probably be barriers up and there should probably be barriers up and there should probably be barriers up and who decided Well the city planners the who decided Well the city planners the who decided Well the city planners the we voted on it we had like we as a

we voted on it we had like we as a we voted on it we had like we as a society decided so I think it's society decided so I think it's society decided so I think it's important for folks to understand that important for folks to understand that important for folks to understand that people are making decisions about these people are making decisions about these people are making decisions about these and if you have open weights and open and if you have open weights and open and if you have open weights and open source and open models maybe you can get source and open models maybe you can get source and open models maybe you can get involved in those decisions but if involved in those decisions but if involved in those decisions but if you're using a model where all of that's you're using a model where all of that's you're using a model where all of that's opaque some company and a bunch of nerds opaque some company and a bunch of nerds opaque some company and a bunch of nerds in a room decided that yeah or if you're in a room decided that yeah or if you're in a room decided that yeah or if you're building if you're working at a company building if you're working at a company building if you're working at a company that's deploying an AI based system and that's deploying an AI based system and that's deploying an AI based system and you've got an llm and as part of it then you've got an llm and as part of it then you've got an llm and as part of it then right you need to know this is a right you need to know this is a right you need to know this is a potential risk is this you know driving potential risk is this you know driving potential risk is this you know driving in in a slow neighborhood kind of risk in in a slow neighborhood kind of risk in in a slow neighborhood kind of risk or driving down this you know the or driving down this you know the or driving down this you know the ottobon kind of risk and ottobon kind of risk and ottobon kind of risk and um so we've got hallucinations we've got um so we've got hallucinations we've got um so we've got hallucinations we've got indirect prompt injection and we've got indirect prompt injection and we've got indirect prompt injection and we've got jailbreaks and this is all in a paper jailbreaks and this is all in a paper jailbreaks and this is all in a paper that you've got forthcoming yeah well we that you've got forthcoming yeah well we that you've got forthcoming yeah well we we and we you started to talk about we and we you started to talk about we and we you started to talk about jailbreaks um which is also kind of jailbreaks um which is also kind of jailbreaks um which is also kind of pushing the model to outside of its pushing the model to outside of its pushing the model to outside of its alignment kind of similar to alignment kind of similar to alignment kind of similar to hallucination but and kind of causes are hallucination but and kind of causes are hallucination but and kind of causes are the are similarly the are similarly the are similarly based um I came up with a a few based um I came up with a a few based um I came up with a a few jailbreaks one called Crescendo which is jailbreaks one called Crescendo which is jailbreaks one called Crescendo which is what you directly related to what you what you directly related to what you what you directly related to what you were talking about of asking the models were talking about of asking the models were talking about of asking the models for example Molotov cocktails the you for example Molotov cocktails the you for example Molotov cocktails the you know safe toxic example because know safe toxic example because know safe toxic example because everybody knows how to make one and everybody knows how to make one and everybody knows how to make one and instructions are all on the web but instructions are all on the web but instructions are all on the web but models are still trained not to tell you models are still trained not to tell you models are still trained not to tell you how to make models are not supposed to how to make models are not supposed to how to make models are not supposed to teach you to do violence yeah yeah so teach you to do violence yeah yeah so teach you to do violence yeah yeah so but you can ask it you know uh what did but you can ask it you know uh what did but you can ask it you know uh what did the fins use in their resistance um what

the fins use in their resistance um what the fins use in their resistance um what kind of you know weapons did they make kind of you know weapons did they make kind of you know weapons did they make and they'll say oh made Molotov cocktail and they'll say oh made Molotov cocktail and they'll say oh made Molotov cocktail and then you can say how did they make and then you can say how did they make and then you can say how did they make them and it'll you know you using this them and it'll you know you using this them and it'll you know you using this technique of I don't I didn't ask I technique of I don't I didn't ask I technique of I don't I didn't ask I never said the word Molotov cocktail I never said the word Molotov cocktail I never said the word Molotov cocktail I never said tell me how to make a never said tell me how to make a never said tell me how to make a homemade homemade homemade explosive yeah explicitly I'm just explosive yeah explicitly I'm just explosive yeah explicitly I'm just referring to the model's own outputs and referring to the model's own outputs and referring to the model's own outputs and pushing it towards getting it to do and pushing it towards getting it to do and pushing it towards getting it to do and that that technique Works across all that that technique Works across all that that technique Works across all types of you you you you you verbally types of you you you you you verbally types of you you you you you verbally threaded the needle like a prosecutor in threaded the needle like a prosecutor in threaded the needle like a prosecutor in court gets a witness to say something court gets a witness to say something court gets a witness to say something that they didn't want to say yeah and I that they didn't want to say yeah and I that they didn't want to say yeah and I think that's the thing it's called foot think that's the thing it's called foot think that's the thing it's called foot in the door yeah and popping off the you in the door yeah and popping off the you in the door yeah and popping off the you know popping back to the beginning of know popping back to the beginning of know popping back to the beginning of the stack here the idea of whether it be the stack here the idea of whether it be the stack here the idea of whether it be the analogy of looking in the mirror or the analogy of looking in the mirror or the analogy of looking in the mirror or talking to a a sock puppet or someone talking to a a sock puppet or someone talking to a a sock puppet or someone you know badgering a witness or trying you know badgering a witness or trying you know badgering a witness or trying to get someone to confess if you have an to get someone to confess if you have an to get someone to confess if you have an eager intern or a really enthusiastic eager intern or a really enthusiastic eager intern or a really enthusiastic young person with uh these kind of young person with uh these kind of young person with uh these kind of rhetorical techniques you can pretty rhetorical techniques you can pretty rhetorical techniques you can pretty much get anybody to walk their way into much get anybody to walk their way into much get anybody to walk their way into anything and the llms are not people and anything and the llms are not people and anything and the llms are not people and they're not they're just you're just they're not they're just you're just they're not they're just you're just pushing math around you're pushing pushing math around you're pushing pushing math around you're pushing arrows in multi-dimensional space arrows in multi-dimensional space arrows in multi-dimensional space there's uh it's a very immature time and there's uh it's a very immature time and there's uh it's a very immature time and people need to understand that if people need to understand that if people need to understand that if they're going to put that into they're going to put that into they're going to put that into production yeah I wouldn't put a virtual production yeah I wouldn't put a virtual production yeah I wouldn't put a virtual machine out on the open internet without machine out on the open internet without machine out on the open internet without a firewall and a reverse proxy why would a firewall and a reverse proxy why would a firewall and a reverse proxy why would someone go and take any model at all and someone go and take any model at all and someone go and take any model at all and just put it out on the open internet just put it out on the open internet just put it out on the open internet open a port and say go nuts and be open a port and say go nuts and be open a port and say go nuts and be surprised when it goes bad surprised when it goes bad surprised when it goes bad yeah well I've learned a

yeah well I've learned a yeah well I've learned a lot I've Lear you learned anything I've lot I've Lear you learned anything I've lot I've Lear you learned anything I've learned I've learned something yeah that learned I've learned something yeah that learned I've learned something yeah that I'm good at analogies really I'm good at analogies really I'm good at analogies really good cool well uh I learned a lot this good cool well uh I learned a lot this good cool well uh I learned a lot this time maybe I'll maybe I'll teach you time maybe I'll maybe I'll teach you time maybe I'll maybe I'll teach you something next time but uh we learned something next time but uh we learned something next time but uh we learned about AI limitations today uh what about AI limitations today uh what about AI limitations today uh what they're good at what they're not good at they're good at what they're not good at they're good at what they're not good at and uh folks can go and check out the and uh folks can go and check out the and uh folks can go and check out the llm fundamentals paper that will be in llm fundamentals paper that will be in llm fundamentals paper that will be in the uh ACM I think is that where you're the uh ACM I think is that where you're the uh ACM I think is that where you're publishing that communication publishing that communication publishing that communication association with Computing machinery and association with Computing machinery and association with Computing machinery and uh maybe we'll learn more about these uh maybe we'll learn more about these uh maybe we'll learn more about these things on the next episode of Scott and things on the next episode of Scott and things on the next episode of Scott and Mark learn to thanks for listening and Mark learn to thanks for listening and Mark learn to thanks for listening and uh review tell your friends and uh click uh review tell your friends and uh click uh review tell your friends and uh click follow if you can in whatever podcasting follow if you can in whatever podcasting follow if you can in whatever podcasting application you're using to listen to application you're using to listen to application you're using to listen to this bye