Inference Engineering with Baseten's Philip Kiely

There's something new from our sponsor There's something new from our sponsor Text Control. For decades, TX Text Text Control. For decades, TX Text Text Control. For decades, TX Text Control's rendering was tied to Windows Control's rendering was tied to Windows Control's rendering was tied to Windows functionality, but now it's a new era. functionality, but now it's a new era. functionality, but now it's a new era. TX Text Control is platform independent. TX Text Control is platform independent. TX Text Control is platform independent. You can run your .NET applications on You can run your .NET applications on You can run your .NET applications on Windows or Linux. Deploy with Docker Windows or Linux. Deploy with Docker Windows or Linux. Deploy with Docker containers, Kubernetes, or host directly containers, Kubernetes, or host directly containers, Kubernetes, or host directly in Azure App Services. Seamlessly in Azure App Services. Seamlessly in Azure App Services. Seamlessly integrate document editing, signing, integrate document editing, signing, integrate document editing, signing, collaboration, and PDF processing into collaboration, and PDF processing into collaboration, and PDF processing into your ASP.NET Core and Angular your ASP.NET Core and Angular your ASP.NET Core and Angular applications. applications. applications. Are you ready to see it in action? Check Are you ready to see it in action? Check Are you ready to see it in action? Check out the live demo at out the live demo at out the live demo at demos.textcontrol.com. >> Hey friends, I'm Scott Hanselman and >> Hey friends, I'm Scott Hanselman and this is another episode of this is another episode of this is another episode of Hanselminutes. And today I'm chatting Hanselminutes. And today I'm chatting Hanselminutes. And today I'm chatting with Philip Kyle. He is the author of with Philip Kyle. He is the author of with Philip Kyle. He is the author of inference engineering and he works at inference engineering and he works at inference engineering and he works at Base 10. You can check it out at Base 10. You can check it out at Base 10. You can check it out at base10.co.

base10.co. base10.co. How's it going, sir? How's it going, sir? How's it going, sir? >> I'm doing great, Scott. It's an honor to >> I'm doing great, Scott. It's an honor to >> I'm doing great, Scott. It's an honor to be here. be here. be here. >> Well, it's cool to hang out with you. >> Well, it's cool to hang out with you. >> Well, it's cool to hang out with you. So, So, So, this book came out of nowhere and you this book came out of nowhere and you this book came out of nowhere and you you you tweeted about it. I saw it. I you you tweeted about it. I saw it. I you you tweeted about it. I saw it. I saw the opening video and then you were saw the opening video and then you were saw the opening video and then you were kind enough to send me a copy, kind enough to send me a copy, kind enough to send me a copy, which I have read and dog-eared and I've which I have read and dog-eared and I've which I have read and dog-eared and I've got a bunch of dog-eared pages here with got a bunch of dog-eared pages here with got a bunch of dog-eared pages here with questions that I want to ask you about. questions that I want to ask you about. questions that I want to ask you about. But I I got to just shout out like But I I got to just shout out like But I I got to just shout out like I There's just so much kind of like slop I There's just so much kind of like slop I There's just so much kind of like slop right now. There's so much AI slop. right now. There's so much AI slop. right now. There's so much AI slop. And then a book shows up that kind of And then a book shows up that kind of And then a book shows up that kind of like is the opposite in the sense of like is the opposite in the sense of like is the opposite in the sense of like it smells good. It's got a really like it smells good. It's got a really like it smells good. It's got a really nice design. Like like I was like the nice design. Like like I was like the nice design. Like like I was like the first thing I thought was like, I hope first thing I thought was like, I hope first thing I thought was like, I hope they didn't AI generate the design. And they didn't AI generate the design. And they didn't AI generate the design. And then you actually shout out the author then you actually shout out the author then you actually shout out the author in the book who did the design custom. in the book who did the design custom. in the book who did the design custom. Like it feels bespoke in a moment where Like it feels bespoke in a moment where Like it feels bespoke in a moment where everything's not bespoke. Was that a everything's not bespoke. Was that a everything's not bespoke. Was that a conscious decision to do something like conscious decision to do something like conscious decision to do something like that? that? that? >> Absolutely. I mean, the thing is, Scott, >> Absolutely. I mean, the thing is, Scott, >> Absolutely. I mean, the thing is, Scott, I love writing books. I've done I've I love writing books. I've done I've I love writing books. I've done I've done it before, but I've never done a done it before, but I've never done a done it before, but I've never done a good job like this. So, it took me a few good job like this. So, it took me a few good job like this. So, it took me a few tries to to get here. So, I wrote I tries to to get here. So, I wrote I tries to to get here. So, I wrote I wrote Influence Engineering because, wrote Influence Engineering because, wrote Influence Engineering because, number one, I love the topic, and number number one, I love the topic, and number number one, I love the topic, and number two, I love writing books. And if I was, two, I love writing books. And if I was, two, I love writing books. And if I was, you know, just saying I mean, you saw in you know, just saying I mean, you saw in you know, just saying I mean, you saw in the launch video, the the very first the launch video, the the very first the launch video, the the very first line of the video was, "Hey, you know, line of the video was, "Hey, you know, line of the video was, "Hey, you know, they made me take it out, but I was they made me take it out, but I was they made me take it out, but I was going to say, 'Hey, Claude, write me a going to say, 'Hey, Claude, write me a going to say, 'Hey, Claude, write me a book.'" I just said, "Hey, hey, write me book.'" I just said, "Hey, hey, write me book.'" I just said, "Hey, hey, write me a book a book a book called Influence Engineering: Make No called Influence Engineering: Make No called Influence Engineering: Make No Mistakes."

Mistakes." Mistakes." It it it it just doesn't work. You just It it it it just doesn't work. You just It it it it just doesn't work. You just can't do it. So, can't do it. So, can't do it. So, >> Well, that's the other thing. >> Well, that's the other thing. >> Well, that's the other thing. We got to we we got to say right off the We got to we we got to say right off the We got to we we got to say right off the bat, when you're writing a book about bat, when you're writing a book about bat, when you're writing a book about AI, you're writing a book about the AI, you're writing a book about the AI, you're writing a book about the technologies that power AI, the first technologies that power AI, the first technologies that power AI, the first question that an interviewer has to ask question that an interviewer has to ask question that an interviewer has to ask is, is, is, "Did you AI slop the book and and have "Did you AI slop the book and and have "Did you AI slop the book and and have the machine write the book for you, or the machine write the book for you, or the machine write the book for you, or is this four years of actually writing is this four years of actually writing is this four years of actually writing hard?" hard?" hard?" >> So, I tried my best to AI slop this >> So, I tried my best to AI slop this >> So, I tried my best to AI slop this because I I came up with the idea for because I I came up with the idea for because I I came up with the idea for this book six months ago. And, you know, this book six months ago. And, you know, this book six months ago. And, you know, if you go talk to a traditional if you go talk to a traditional if you go talk to a traditional publisher, which I did, they'll tell publisher, which I did, they'll tell publisher, which I did, they'll tell you, "Once you give me a manuscript, you, "Once you give me a manuscript, you, "Once you give me a manuscript, it'll take 12 to 18 months to get this it'll take 12 to 18 months to get this it'll take 12 to 18 months to get this thing out into the world." And I knew thing out into the world." And I knew thing out into the world." And I knew that I wanted to go from idea to that I wanted to go from idea to that I wanted to go from idea to finished hardcovers all over the world finished hardcovers all over the world finished hardcovers all over the world in six months. in six months. in six months. And so, anything that I could do to make And so, anything that I could do to make And so, anything that I could do to make that faster, I wanted to do. So, 100% I that faster, I wanted to do. So, 100% I that faster, I wanted to do. So, 100% I tried. I said, you know, "Hey, can can tried. I said, you know, "Hey, can can tried. I said, you know, "Hey, can can you come up with the outline? Can you you come up with the outline? Can you you come up with the outline? Can you write this chapter, write that chapter?"

write this chapter, write that chapter?" write this chapter, write that chapter?" But, the quality was just completely But, the quality was just completely But, the quality was just completely unusable. unusable. unusable. There were a few places where I was able There were a few places where I was able There were a few places where I was able to use AI. So, number one, to use AI. So, number one, to use AI. So, number one, by the way, Scott, like have you done by the way, Scott, like have you done by the way, Scott, like have you done any long-form writing? I assume I assume any long-form writing? I assume I assume any long-form writing? I assume I assume you haven't and you you haven't and you you haven't and you >> No, I yeah, I've written a couple of >> No, I yeah, I've written a couple of >> No, I yeah, I've written a couple of books. I've got it and and none of them books. I've got it and and none of them books. I've got it and and none of them used AI, and some of them are almost a used AI, and some of them are almost a used AI, and some of them are almost a thousand pages long and it's mostly thousand pages long and it's mostly thousand pages long and it's mostly know, it's code samples like know, it's code samples like know, it's code samples like I I mean like this is the thing like I I mean like this is the thing like I I mean like this is the thing like ironically we're talking about a book ironically we're talking about a book ironically we're talking about a book about AI. It doesn't feel like there's about AI. It doesn't feel like there's about AI. It doesn't feel like there's any AI. I read the book. I got the book any AI. I read the book. I got the book any AI. I read the book. I got the book marked up. None of my books are used AI marked up. None of my books are used AI marked up. None of my books are used AI because it's just it doesn't hold a because it's just it doesn't hold a because it's just it doesn't hold a through line. It doesn't hold a topic through line. It doesn't hold a topic through line. It doesn't hold a topic long enough. long enough. long enough. >> Exactly. So let me tell you how to use >> Exactly. So let me tell you how to use >> Exactly. So let me tell you how to use AI if you're writing a book because yeah AI if you're writing a book because yeah AI if you're writing a book because yeah for the most part for the most part I for the most part for the most part I for the most part for the most part I didn't. Um if you look at for example didn't. Um if you look at for example didn't. Um if you look at for example chapter four chapter four chapter four section the figure 4.1 this guy right section the figure 4.1 this guy right section the figure 4.1 this guy right here. It's a six line code a kernel. here. It's a six line code a kernel. here. It's a six line code a kernel. >> Okay. >> Okay. >> Okay. Got it. Got it. Got it. >> I asked for an example code a kernel. >> I asked for an example code a kernel. >> I asked for an example code a kernel. That saved me 30 minutes, you know.

That saved me 30 minutes, you know. That saved me 30 minutes, you know. >> I see. So the code some of the code >> I see. So the code some of the code >> I see. So the code some of the code might be might be might be >> You know, you can it's it's small it's >> You know, you can it's it's small it's >> You know, you can it's it's small it's small pieces like that that save 30 small pieces like that that save 30 small pieces like that that save 30 minutes there an hour there that you can minutes there an hour there that you can minutes there an hour there that you can verify yourself and make sure they're verify yourself and make sure they're verify yourself and make sure they're good. It's you know, good. It's you know, good. It's you know, like I said you've you've written before like I said you've you've written before like I said you've you've written before you know how hard it is to look at the you know how hard it is to look at the you know how hard it is to look at the blank page. It's coming up with a few blank page. It's coming up with a few blank page. It's coming up with a few bad ideas so that I can throw those away bad ideas so that I can throw those away bad ideas so that I can throw those away and be inspired to create a good one. and be inspired to create a good one. and be inspired to create a good one. It's taking your outline or taking your It's taking your outline or taking your It's taking your outline or taking your sections and checking them for factual sections and checking them for factual sections and checking them for factual accuracy, checking them for accuracy, checking them for accuracy, checking them for for completeness, but you can't just for completeness, but you can't just for completeness, but you can't just take the outputs and say oh well every take the outputs and say oh well every take the outputs and say oh well every fact check it just gave me means all of fact check it just gave me means all of fact check it just gave me means all of this is wrong. No, that's just your this is wrong. No, that's just your this is wrong. No, that's just your to-do list to go do manually yourself. to-do list to go do manually yourself. to-do list to go do manually yourself. And then if you look in the back like And then if you look in the back like And then if you look in the back like every time I write a book I write custom every time I write a book I write custom every time I write a book I write custom software to make writing and laying out software to make writing and laying out software to make writing and laying out the book easier. So for example like the book easier. So for example like the book easier. So for example like these QR codes and the layout of that. these QR codes and the layout of that. these QR codes and the layout of that. So So So I didn't you know AI generate that I didn't you know AI generate that I didn't you know AI generate that because you can't it's not going to be because you can't it's not going to be because you can't it's not going to be good, but I had Koso write me a script good, but I had Koso write me a script good, but I had Koso write me a script to do that. In appendix A it's to do that. In appendix A it's to do that. In appendix A it's alphabetized, you know. I actually tried alphabetized, you know. I actually tried alphabetized, you know. I actually tried pasting the entirety of appendix A into pasting the entirety of appendix A into pasting the entirety of appendix A into chat GPT and I said, "Hey, alphabetize chat GPT and I said, "Hey, alphabetize chat GPT and I said, "Hey, alphabetize this." And it it got it wrong. Um and this." And it it got it wrong. Um and this." And it it got it wrong. Um and then then then I said, "Hey Koso, write me a script I said, "Hey Koso, write me a script I said, "Hey Koso, write me a script that will alphabetize this." And it made that will alphabetize this." And it made that will alphabetize this." And it made 20 lines of Python and it alphabetized 20 lines of Python and it alphabetized 20 lines of Python and it alphabetized it perfectly.

it perfectly. it perfectly. So, the the main way that I used AI to So, the the main way that I used AI to So, the the main way that I used AI to write this book was not the actual write this book was not the actual write this book was not the actual words, was not the diagrams. The words words, was not the diagrams. The words words, was not the diagrams. The words are by me, the diagrams are by human are by me, the diagrams are by human are by me, the diagrams are by human artists and human designers. It's to artists and human designers. It's to artists and human designers. It's to write the code that makes everything write the code that makes everything write the code that makes everything else else else work faster in the writing, editing, and work faster in the writing, editing, and work faster in the writing, editing, and publishing process. publishing process. publishing process. >> And to And I feel like, you know, in a >> And to And I feel like, you know, in a >> And to And I feel like, you know, in a moment where there's just so much crap moment where there's just so much crap moment where there's just so much crap being generated, I think that is a good being generated, I think that is a good being generated, I think that is a good example of how you want to use AI. I example of how you want to use AI. I example of how you want to use AI. I always use the word toil. always use the word toil. always use the word toil. If it's tedious and it's toil, then AI If it's tedious and it's toil, then AI If it's tedious and it's toil, then AI will do it. Like you say, generating QR will do it. Like you say, generating QR will do it. Like you say, generating QR codes or writing scripts to to to codes or writing scripts to to to codes or writing scripts to to to alphabetize things. But if it's alphabetize things. But if it's alphabetize things. But if it's something like like the like the cover. something like like the like the cover. something like like the like the cover. Like I immediately looked at the cover Like I immediately looked at the cover Like I immediately looked at the cover and I was like, "This is really cool." and I was like, "This is really cool." and I was like, "This is really cool." And I wanted to see the human name of And I wanted to see the human name of And I wanted to see the human name of the person that wrote the cover, you the person that wrote the cover, you the person that wrote the cover, you know what I mean? And I like this you know what I mean? And I like this you know what I mean? And I like this you know, it's like cover by Luke de Haas, know, it's like cover by Luke de Haas, know, it's like cover by Luke de Haas, you know? And shout out to Luke. you know? And shout out to Luke. you know? And shout out to Luke. And I don't know, it's just like I want And I don't know, it's just like I want And I don't know, it's just like I want people want both and you want the hard people want both and you want the hard people want both and you want the hard parts that humans work on to to feel parts that humans work on to to feel parts that humans work on to to feel hard and I can feel that you were hard and I can feel that you were hard and I can feel that you were probably up late probably up late probably up late >> [laughter] >> [laughter] >> [laughter] >> working on this book. Are you okay? Did >> working on this book. Are you okay? Did >> working on this book. Are you okay? Did you hurt yourself in the writing of the you hurt yourself in the writing of the you hurt yourself in the writing of the book?

book? book? >> There were there were a lot of very long >> There were there were a lot of very long >> There were there were a lot of very long writing days. Um the the actual drafting writing days. Um the the actual drafting writing days. Um the the actual drafting piece was about a 6-week sprint where 4 piece was about a 6-week sprint where 4 piece was about a 6-week sprint where 4 days a week I would wake up at 5:00 in days a week I would wake up at 5:00 in days a week I would wake up at 5:00 in the morning and write until I fell the morning and write until I fell the morning and write until I fell asleep and then the other 3 days I'd go asleep and then the other 3 days I'd go asleep and then the other 3 days I'd go to work and do my actual job. Um to work and do my actual job. Um to work and do my actual job. Um So, yeah. And And of course, shout out So, yeah. And And of course, shout out So, yeah. And And of course, shout out to Luke. Um he's the the one of the to Luke. Um he's the the one of the to Luke. Um he's the the one of the brand designers that we have at Basecamp brand designers that we have at Basecamp brand designers that we have at Basecamp and and he did a fantastic job on the and and he did a fantastic job on the and and he did a fantastic job on the cover and setting the design language cover and setting the design language cover and setting the design language for the interior. There was in chapter for the interior. There was in chapter for the interior. There was in chapter one this little diagram of some football one this little diagram of some football one this little diagram of some football players. That's my favorite part in the players. That's my favorite part in the players. That's my favorite part in the entire book. Um yeah, it it was I think entire book. Um yeah, it it was I think entire book. Um yeah, it it was I think this project was a great example of this project was a great example of this project was a great example of collaboration between multiple humans collaboration between multiple humans collaboration between multiple humans and between AI tools because writing a and between AI tools because writing a and between AI tools because writing a book is toil. This to overuse a book is toil. This to overuse a book is toil. This to overuse a metaphor, it's like running a marathon. metaphor, it's like running a marathon. metaphor, it's like running a marathon. No individual step is that hard. The No individual step is that hard. The No individual step is that hard. The hardest part is taking all the steps hardest part is taking all the steps hardest part is taking all the steps back to back. Not every piece of toil back to back. Not every piece of toil back to back. Not every piece of toil can be can be automated. For example, can be can be automated. For example, can be can be automated. For example, that book you're holding in your hand is that book you're holding in your hand is that book you're holding in your hand is signed. Do you know how long it takes to signed. Do you know how long it takes to signed. Do you know how long it takes to sign 300 hard covers? Cuz I just found sign 300 hard covers? Cuz I just found sign 300 hard covers? Cuz I just found out.

out. out. >> I I did note your hands were a little >> I I did note your hands were a little >> I I did note your hands were a little shaky when you when you when you signed shaky when you when you when you signed shaky when you when you when you signed when you signed the book. when you signed the book. when you signed the book. >> My handwriting's just that bad, Scott. >> My handwriting's just that bad, Scott. >> My handwriting's just that bad, Scott. That's just how it is. That's just how it is. That's just how it is. >> Um the other thing that I think is worth >> Um the other thing that I think is worth >> Um the other thing that I think is worth noting and I want to shout out like noting and I want to shout out like noting and I want to shout out like people who might buy the book or might people who might buy the book or might people who might buy the book or might think about getting into the digital think about getting into the digital think about getting into the digital download or a paper copy is that this is download or a paper copy is that this is download or a paper copy is that this is a dense book. There's a lot of garbage a dense book. There's a lot of garbage a dense book. There's a lot of garbage out there that is just stuff you can out there that is just stuff you can out there that is just stuff you can learn online. learn online. learn online. Uh what I appreciated about your effort Uh what I appreciated about your effort Uh what I appreciated about your effort was that was that was that I had to work pretty hard I had to work pretty hard I had to work pretty hard at this book. Like I had to at this book. Like I had to at this book. Like I had to uh at points I had to stop, look stuff uh at points I had to stop, look stuff uh at points I had to stop, look stuff up. Um there were a couple of times up. Um there were a couple of times up. Um there were a couple of times where I did chat GPT voice mode and where I did chat GPT voice mode and where I did chat GPT voice mode and asked for to understand stuff better. asked for to understand stuff better. asked for to understand stuff better. Um it is it is dense and it seems Um it is it is dense and it seems Um it is it is dense and it seems surprising to me that a company surprising to me that a company surprising to me that a company that makes AI easier would write a book that makes AI easier would write a book that makes AI easier would write a book about the hard stuff. How did you decide about the hard stuff. How did you decide about the hard stuff. How did you decide to write a hard book rather than to like to write a hard book rather than to like to write a hard book rather than to like make a puff piece? make a puff piece? make a puff piece? >> Well, for one thing to me this is uh >> Well, for one thing to me this is uh >> Well, for one thing to me this is uh this is kind of not the hard stuff. The this is kind of not the hard stuff. The this is kind of not the hard stuff. The The hard stuff is the model performance The hard stuff is the model performance The hard stuff is the model performance [laughter] engineers are doing every day [laughter] engineers are doing every day [laughter] engineers are doing every day and the infrastructure engineers, you and the infrastructure engineers, you and the infrastructure engineers, you know, I'm just I'm just a a friendly dev know, I'm just I'm just a a friendly dev know, I'm just I'm just a a friendly dev well with a with an undergrad CS degree.

well with a with an undergrad CS degree. well with a with an undergrad CS degree. Um no, I I I get what you mean. That's Um no, I I I get what you mean. That's Um no, I I I get what you mean. That's another reason that I couldn't use very another reason that I couldn't use very another reason that I couldn't use very much AI to write this. It's like much AI to write this. It's like much AI to write this. It's like the the LLMs of today just don't know the the LLMs of today just don't know the the LLMs of today just don't know this material very well because it's so this material very well because it's so this material very well because it's so new. new. new. But But But my goal for this book, number one, was my goal for this book, number one, was my goal for this book, number one, was that anyone would be able to pick it up that anyone would be able to pick it up that anyone would be able to pick it up and read the first 20 pages and and read the first 20 pages and and read the first 20 pages and understand my vision for where the understand my vision for where the understand my vision for where the future of AI is going. future of AI is going. future of AI is going. And then, number two, if they, you know, And then, number two, if they, you know, And then, number two, if they, you know, make 20 pages and they're really excited make 20 pages and they're really excited make 20 pages and they're really excited about the topic, that, you know, any about the topic, that, you know, any about the topic, that, you know, any smart person with a decent technical smart person with a decent technical smart person with a decent technical background is is going to be able to background is is going to be able to background is is going to be able to make it through the rest. make it through the rest. make it through the rest. And that meant making some trade-offs. And that meant making some trade-offs. And that meant making some trade-offs. You know, I I've definitely heard one of You know, I I've definitely heard one of You know, I I've definitely heard one of the, I think, most valid criticisms of the, I think, most valid criticisms of the, I think, most valid criticisms of this book is that it's a mile wide and this book is that it's a mile wide and this book is that it's a mile wide and an inch deep. It's really much a survey an inch deep. It's really much a survey an inch deep. It's really much a survey of all of these technologies. It doesn't of all of these technologies. It doesn't of all of these technologies. It doesn't necessarily teach you how to go hands-on necessarily teach you how to go hands-on necessarily teach you how to go hands-on and step-by-step do something. It's It's and step-by-step do something. It's It's and step-by-step do something. It's It's not one of your 1,000 page books with a not one of your 1,000 page books with a not one of your 1,000 page books with a ton of code samples and a ton of ton of code samples and a ton of ton of code samples and a ton of hands-on guidance. And the reason I made hands-on guidance. And the reason I made hands-on guidance. And the reason I made that decision is partially just because that decision is partially just because that decision is partially just because I wanted a book that I could write very I wanted a book that I could write very I wanted a book that I could write very quickly. And partially [clears throat] quickly. And partially [clears throat] quickly. And partially [clears throat] because the goal of this book is more to because the goal of this book is more to because the goal of this book is more to teach someone what questions to ask teach someone what questions to ask teach someone what questions to ask rather than give them all of the rather than give them all of the rather than give them all of the answers. So, yeah, you know, it it is a answers. So, yeah, you know, it it is a answers. So, yeah, you know, it it is a dense book. It's a hard book. But like, dense book. It's a hard book. But like, dense book. It's a hard book. But like, we have all of the sales people when we have all of the sales people when we have all of the sales people when they join Base 10, they they read this they join Base 10, they they read this they join Base 10, they they read this and and they get through it. So, there's

and and they get through it. So, there's and and they get through it. So, there's definitely, I think, a level of definitely, I think, a level of definitely, I think, a level of accessibility to the content that I accessibility to the content that I accessibility to the content that I prioritized. prioritized. prioritized. >> Interesting. Yeah, I mean, I did >> Interesting. Yeah, I mean, I did >> Interesting. Yeah, I mean, I did survey seems light. To say that it is a survey seems light. To say that it is a survey seems light. To say that it is a survey survey survey I mean, it is it is an overview, surely, I mean, it is it is an overview, surely, I mean, it is it is an overview, surely, of everything that is going on right now of everything that is going on right now of everything that is going on right now at this moment in time. And I'm sure at this moment in time. And I'm sure at this moment in time. And I'm sure that in 2 years, you should probably that in 2 years, you should probably that in 2 years, you should probably start working on or maybe a year, you start working on or maybe a year, you start working on or maybe a year, you start working on on second edition. start working on on second edition. start working on on second edition. >> Absolutely. >> Absolutely. >> Absolutely. >> Right. Because there that's the other >> Right. Because there that's the other >> Right. Because there that's the other irony of like having a physical book uh irony of like having a physical book uh irony of like having a physical book uh with actual pages and actual dead trees with actual pages and actual dead trees with actual pages and actual dead trees means that uh you know, how much of this means that uh you know, how much of this means that uh you know, how much of this is out of date? Not a lot. It's actually is out of date? Not a lot. It's actually is out of date? Not a lot. It's actually pretty pretty pretty fresh. pretty pretty pretty fresh. pretty pretty pretty fresh. But um it did ask it did give me a lot But um it did ask it did give me a lot But um it did ask it did give me a lot of questions to ask. And it's funny that of questions to ask. And it's funny that of questions to ask. And it's funny that you mentioned that because I'm looking you mentioned that because I'm looking you mentioned that because I'm looking here on my first question. here on my first question. here on my first question. I made it I made it I made it Let me see here. 41 pages in before I Let me see here. 41 pages in before I Let me see here. 41 pages in before I started dog-earing and looking stuff up. started dog-earing and looking stuff up. started dog-earing and looking stuff up. So, you said people If If smart people So, you said people If If smart people So, you said people If If smart people can make it 20 pages in, that's a good can make it 20 pages in, that's a good can make it 20 pages in, that's a good sign. I made it 41. sign. I made it 41. sign. I made it 41. >> Well, there you go. >> Well, there you go. >> Well, there you go. >> 42 pages. So, I feel somewhat validated >> 42 pages. So, I feel somewhat validated >> 42 pages. So, I feel somewhat validated uh uh uh about this.

about this. about this. Um Did Did you feel uncomfortable that Um Did Did you feel uncomfortable that Um Did Did you feel uncomfortable that you don't have a PhD in in machine you don't have a PhD in in machine you don't have a PhD in in machine learning and deep learning? Did you Did learning and deep learning? Did you Did learning and deep learning? Did you Did you have you have you have Did you Did you say like, "Uh should I Did you Did you say like, "Uh should I Did you Did you say like, "Uh should I be the one that writes the book?" How be the one that writes the book?" How be the one that writes the book?" How did you deal with that? Cuz I always did you deal with that? Cuz I always did you deal with that? Cuz I always feel like an impostor when I'm writing feel like an impostor when I'm writing feel like an impostor when I'm writing technical books. technical books. technical books. >> I am blessed with a complete absence of >> I am blessed with a complete absence of >> I am blessed with a complete absence of impostor syndrome. impostor syndrome. impostor syndrome. >> Oh, wow. You're the first. That's >> Oh, wow. You're the first. That's >> Oh, wow. You're the first. That's amazing. Tell us your ways. amazing. Tell us your ways. amazing. Tell us your ways. >> Perhaps some people in my life would uh >> Perhaps some people in my life would uh >> Perhaps some people in my life would uh would would tell me that that a little would would tell me that that a little would would tell me that that a little bit of it might be good for me. bit of it might be good for me. bit of it might be good for me. Um but Um but Um but I think that there's a very academic I think that there's a very academic I think that there's a very academic take on AI that starts, you know, in the take on AI that starts, you know, in the take on AI that starts, you know, in the early days and defines what a perceptron early days and defines what a perceptron early days and defines what a perceptron is and teaches you what a convolutional is and teaches you what a convolutional is and teaches you what a convolutional neural network is and all that kind of neural network is and all that kind of neural network is and all that kind of stuff. And stuff. And stuff. And I entered the AI industry in 2022, I entered the AI industry in 2022, I entered the AI industry in 2022, January of 2022. About 10 months before January of 2022. About 10 months before January of 2022. About 10 months before ChatGPT became a huge thing. ChatGPT became a huge thing. ChatGPT became a huge thing. >> Mhm. >> Mhm. >> Mhm. >> And I didn't know any of that stuff.

>> And I didn't know any of that stuff. >> And I didn't know any of that stuff. And that hasn't stopped me from doing my And that hasn't stopped me from doing my And that hasn't stopped me from doing my job today. I'm sure all of that stuff is job today. I'm sure all of that stuff is job today. I'm sure all of that stuff is very valuable and I've enjoyed learning, very valuable and I've enjoyed learning, very valuable and I've enjoyed learning, you know, bits and pieces of it along you know, bits and pieces of it along you know, bits and pieces of it along the way. the way. the way. But I believe, and this is part of the But I believe, and this is part of the But I believe, and this is part of the thesis of of this project, that you can thesis of of this project, that you can thesis of of this project, that you can make a valuable contribution to the AI make a valuable contribution to the AI make a valuable contribution to the AI industry without a incredibly rigorous industry without a incredibly rigorous industry without a incredibly rigorous academic backgrounds. Um and that, you academic backgrounds. Um and that, you academic backgrounds. Um and that, you know, the the technologies of today are know, the the technologies of today are know, the the technologies of today are so new and moving so fast that in some so new and moving so fast that in some so new and moving so fast that in some ways it's an advantage to not know what ways it's an advantage to not know what ways it's an advantage to not know what you're doing because at least this way you're doing because at least this way you're doing because at least this way you don't have any bad habits to break. you don't have any bad habits to break. you don't have any bad habits to break. >> [snorts] >> [snorts] >> [snorts] >> Yeah, that's interesting. It's funny >> Yeah, that's interesting. It's funny >> Yeah, that's interesting. It's funny that you mention that because I've I've that you mention that because I've I've that you mention that because I've I've talked about what it feels like to be talked about what it feels like to be talked about what it feels like to be towards the other side of your career, towards the other side of your career, towards the other side of your career, the end of your career. So, one could the end of your career. So, one could the end of your career. So, one could argue that like you're in the opening 10 argue that like you're in the opening 10 argue that like you're in the opening 10 years of your career and I'm in the years of your career and I'm in the years of your career and I'm in the latter 10 of mine. So then, should we, latter 10 of mine. So then, should we, latter 10 of mine. So then, should we, you know, should we shouldn't gatekeep you know, should we shouldn't gatekeep you know, should we shouldn't gatekeep this stuff just because someone's old or this stuff just because someone's old or this stuff just because someone's old or someone's young or someone came in at someone's young or someone came in at someone's young or someone came in at the beginning of the hockey stick growth the beginning of the hockey stick growth the beginning of the hockey stick growth within inference or someone, uh you within inference or someone, uh you within inference or someone, uh you know, started it at MIT or worked on the know, started it at MIT or worked on the know, started it at MIT or worked on the project in the '70s with so-and-so. So, project in the '70s with so-and-so. So, project in the '70s with so-and-so. So, you're right. Like young people with you're right. Like young people with you're right. Like young people with fresh ideas and fresh perspectives are fresh ideas and fresh perspectives are fresh ideas and fresh perspectives are what's going to what's going to what's going to actually do the work and make this thing actually do the work and make this thing actually do the work and make this thing uh make this thing useful.

uh make this thing useful. uh make this thing useful. So, one of the questions I wanted to ask So, one of the questions I wanted to ask So, one of the questions I wanted to ask about usefulness is this talks about about usefulness is this talks about about usefulness is this talks about hardware, talks about software, it talks hardware, talks about software, it talks hardware, talks about software, it talks about I can say an overview of all the about I can say an overview of all the about I can say an overview of all the different GPUs, how they work, different GPUs, how they work, different GPUs, how they work, performance profiling, but underneath it performance profiling, but underneath it performance profiling, but underneath it is a reminder that like none of this is is a reminder that like none of this is is a reminder that like none of this is anything unless it actually does anything unless it actually does anything unless it actually does interesting and helpful work. interesting and helpful work. interesting and helpful work. >> [snorts] >> [snorts] >> [snorts] >> Do you Do you think about the >> Do you Do you think about the >> Do you Do you think about the what can be done with AI because we what can be done with AI because we what can be done with AI because we opened this podcast by saying it can't opened this podcast by saying it can't opened this podcast by saying it can't write a book and I don't know if I want write a book and I don't know if I want write a book and I don't know if I want it to write a book. Like I'm not it to write a book. Like I'm not it to write a book. Like I'm not interested in AI slop, I'm interested in interested in AI slop, I'm interested in interested in AI slop, I'm interested in AI, to our point, doing toil. So, I'm AI, to our point, doing toil. So, I'm AI, to our point, doing toil. So, I'm curious about your personal opinion. curious about your personal opinion. curious about your personal opinion. Like here we are with a book on AI Like here we are with a book on AI Like here we are with a book on AI inference, but it the book couldn't inference, but it the book couldn't inference, but it the book couldn't today be written by an AI and be any today be written by an AI and be any today be written by an AI and be any good. good. good. >> I think very little about vertical AI >> I think very little about vertical AI >> I think very little about vertical AI applications, actually. I mostly spend applications, actually. I mostly spend applications, actually. I mostly spend my I because I spend so much of my day my I because I spend so much of my day my I because I spend so much of my day taking other people's medical taking other people's medical taking other people's medical applications and helping them figure out applications and helping them figure out applications and helping them figure out how to make it twice as fast and half as how to make it twice as fast and half as how to make it twice as fast and half as expensive. expensive. expensive. So, I will say that, So, I will say that, So, I will say that, you know, as a software engineer off and you know, as a software engineer off and you know, as a software engineer off and on throughout my career, obviously the on throughout my career, obviously the on throughout my career, obviously the past couple months have been very past couple months have been very past couple months have been very exciting and exciting and exciting and I feel like I have regained the ability I feel like I have regained the ability I feel like I have regained the ability to ship production code because there's to ship production code because there's to ship production code because there's some there's some trade-offs in in some there's some trade-offs in in some there's some trade-offs in in coming into more of the content space, coming into more of the content space, coming into more of the content space, coming into coming into coming into taking 6 months and writing a book and taking 6 months and writing a book and taking 6 months and writing a book and in that your your other skills can can

in that your your other skills can can in that your your other skills can can atrophy a little bit. So, atrophy a little bit. So, atrophy a little bit. So, I find I find I find that I've been able to that I've been able to that I've been able to augment the sort of missing capabilities augment the sort of missing capabilities augment the sort of missing capabilities and ship the the stuff that I'm really and ship the the stuff that I'm really and ship the the stuff that I'm really excited about, no matter no matter where excited about, no matter no matter where excited about, no matter no matter where it is. Another thing is, you know, video it is. Another thing is, you know, video it is. Another thing is, you know, video editing. I do a lot of video editing editing. I do a lot of video editing editing. I do a lot of video editing with Descript because I'm not a fast or with Descript because I'm not a fast or with Descript because I'm not a fast or skilled editor. If anyone had a chance skilled editor. If anyone had a chance skilled editor. If anyone had a chance to watch my college YouTube videos to watch my college YouTube videos to watch my college YouTube videos before I put took those down a few years before I put took those down a few years before I put took those down a few years ago, ago, ago, they they would certainly know that I'm they they would certainly know that I'm they they would certainly know that I'm no Casey Neistat. no Casey Neistat. no Casey Neistat. >> Mhm. >> Mhm. >> Mhm. >> That's kind of the vision that I have >> That's kind of the vision that I have >> That's kind of the vision that I have for this space. Doesn't particularly for this space. Doesn't particularly for this space. Doesn't particularly matter what my vision is. It matters the matter what my vision is. It matters the matter what my vision is. It matters the vision of the people who are actually vision of the people who are actually vision of the people who are actually building the tools, not just making them building the tools, not just making them building the tools, not just making them faster. faster. faster. But my my vision and my hope is that I'm But my my vision and my hope is that I'm But my my vision and my hope is that I'm going to be able to keep doing the going to be able to keep doing the going to be able to keep doing the things that I really love and outsource things that I really love and outsource things that I really love and outsource the pieces that I am not so good at and the pieces that I am not so good at and the pieces that I am not so good at and create things that otherwise I just create things that otherwise I just create things that otherwise I just would have left on my to-do list.

would have left on my to-do list. would have left on my to-do list. >> That's a that's a healthy attitude. I >> That's a that's a healthy attitude. I >> That's a that's a healthy attitude. I appreciate that. appreciate that. appreciate that. Like you Like you Like you you're staying in your lane and you're you're staying in your lane and you're you're staying in your lane and you're apprecia- you you might dip your toes in apprecia- you you might dip your toes in apprecia- you you might dip your toes in another lane, but you know what you're another lane, but you know what you're another lane, but you know what you're good at and you stay away from the stuff good at and you stay away from the stuff good at and you stay away from the stuff that you're not. that you're not. that you're not. So, Base 10 is about speed, right? Like So, Base 10 is about speed, right? Like So, Base 10 is about speed, right? Like it's about making it better. And a lot it's about making it better. And a lot it's about making it better. And a lot of people concerned, myself I'm of people concerned, myself I'm of people concerned, myself I'm concerned about, you know, concerned about, you know, concerned about, you know, wasting energy on on AI and you spend a wasting energy on on AI and you spend a wasting energy on on AI and you spend a lot of time in the book talking about lot of time in the book talking about lot of time in the book talking about performance. There's a ton like you you performance. There's a ton like you you performance. There's a ton like you you start right off the bat about like start right off the bat about like start right off the bat about like they're already making it faster. You they're already making it faster. You they're already making it faster. You talk about the difference between time talk about the difference between time talk about the difference between time to first token versus throughput. Um the to first token versus throughput. Um the to first token versus throughput. Um the the the whole thing is about throughput the the whole thing is about throughput the the whole thing is about throughput on honestly, if there was like a on honestly, if there was like a on honestly, if there was like a a through line through the book, it's a through line through the book, it's a through line through the book, it's like it's getting faster and it's going like it's getting faster and it's going like it's getting faster and it's going to get faster. How much faster do you to get faster. How much faster do you to get faster. How much faster do you think it can actually get and it's not think it can actually get and it's not think it can actually get and it's not going to be good in the sense of we'll going to be good in the sense of we'll going to be good in the sense of we'll waste less energy and it won't cost this waste less energy and it won't cost this waste less energy and it won't cost this many teaspoons of water to ask a many teaspoons of water to ask a many teaspoons of water to ask a question of a of a model. question of a of a model. question of a of a model. >> So, with performance >> So, with performance >> So, with performance there's a couple ways to think about it.

there's a couple ways to think about it. there's a couple ways to think about it. It's a multi-variable optimization It's a multi-variable optimization It's a multi-variable optimization problem. But let's think just about problem. But let's think just about problem. But let's think just about three variables right now. Speed, cost, three variables right now. Speed, cost, three variables right now. Speed, cost, cost and throughput are the same thing, cost and throughput are the same thing, cost and throughput are the same thing, um and quality, the the correctness of um and quality, the the correctness of um and quality, the the correctness of of the output. of the output. of the output. So, quality is fixed at the model level. So, quality is fixed at the model level. So, quality is fixed at the model level. So, let's take the model as fixed for a So, let's take the model as fixed for a So, let's take the model as fixed for a second. So, now we only have to think second. So, now we only have to think second. So, now we only have to think about two variables, which are speed and about two variables, which are speed and about two variables, which are speed and cost or throughput. cost or throughput. cost or throughput. And you can plot an efficient frontier And you can plot an efficient frontier And you can plot an efficient frontier where on one axis you have speed, one where on one axis you have speed, one where on one axis you have speed, one axis you have cost, you can be maximally axis you have cost, you can be maximally axis you have cost, you can be maximally fast at a very high cost, you can be fast at a very high cost, you can be fast at a very high cost, you can be maximally cheap, but you're not going to maximally cheap, but you're not going to maximally cheap, but you're not going to be very fast, and then there's an be very fast, and then there's an be very fast, and then there's an efficient frontier curve between those efficient frontier curve between those efficient frontier curve between those two. two. two. Some model performance work is just Some model performance work is just Some model performance work is just figuring out where that efficient figuring out where that efficient figuring out where that efficient frontier is and helping you adjust the frontier is and helping you adjust the frontier is and helping you adjust the knobs and dials within your influence knobs and dials within your influence knobs and dials within your influence engine and whatnot to get to the place engine and whatnot to get to the place engine and whatnot to get to the place that you want to go on that frontier.

that you want to go on that frontier. that you want to go on that frontier. And then some performance work, more on And then some performance work, more on And then some performance work, more on the research side, is pushing that the research side, is pushing that the research side, is pushing that frontier out. And when you do that, when frontier out. And when you do that, when frontier out. And when you do that, when you create some kind of optimization you create some kind of optimization you create some kind of optimization that unlocks better performance, you can that unlocks better performance, you can that unlocks better performance, you can choose to move along move along the new choose to move along move along the new choose to move along move along the new frontier to either more speed at the frontier to either more speed at the frontier to either more speed at the same cost or um or less cost at the same same cost or um or less cost at the same same cost or um or less cost at the same speed. speed. speed. So, So, So, then you add back the quality access, then you add back the quality access, then you add back the quality access, suddenly your efficient frontier becomes suddenly your efficient frontier becomes suddenly your efficient frontier becomes a sphere instead of a curve line, and as a sphere instead of a curve line, and as a sphere instead of a curve line, and as you either fine-tune a small open-source you either fine-tune a small open-source you either fine-tune a small open-source model to replace a large model, get a model to replace a large model, get a model to replace a large model, get a new performance frontier, find a spot new performance frontier, find a spot new performance frontier, find a spot along that, um you you ultimately, like along that, um you you ultimately, like along that, um you you ultimately, like as this this research continues, as this this research continues, as this this research continues, you're able to push out and out your you're able to push out and out your you're able to push out and out your options around speed, cost, and quality. Again, kind of staying in my lane, like Again, kind of staying in my lane, like what I'm focused on is making the sphere what I'm focused on is making the sphere what I'm focused on is making the sphere as as big as possible, making it so that as as big as possible, making it so that as as big as possible, making it so that you have as much you have as much you have as much >> [snorts] >> [snorts] >> [snorts] >> sort of performance as a sort of >> sort of performance as a sort of >> sort of performance as a sort of a performance as a a performance as a a performance as a vague and term to spend in these vague and term to spend in these vague and term to spend in these tradeoffs.

tradeoffs. tradeoffs. Where does that actually go? You know, Where does that actually go? You know, Where does that actually go? You know, we definitely see with inference getting we definitely see with inference getting we definitely see with inference getting cheaper and inference getting faster cheaper and inference getting faster cheaper and inference getting faster that the same amount of traffic that you that the same amount of traffic that you that the same amount of traffic that you maybe had 6 months ago, you can now maybe had 6 months ago, you can now maybe had 6 months ago, you can now serve with a fraction of the hardware, serve with a fraction of the hardware, serve with a fraction of the hardware, or you can now serve for a fraction of or you can now serve for a fraction of or you can now serve for a fraction of the cost. You can now serve at a much the cost. You can now serve at a much the cost. You can now serve at a much better speed. better speed. better speed. But, But, But, that is overshadowed by increases in that is overshadowed by increases in that is overshadowed by increases in demand. Um it's funny because a lot of demand. Um it's funny because a lot of demand. Um it's funny because a lot of the time we spend as a company is like the time we spend as a company is like the time we spend as a company is like trying to figure out how to get our trying to figure out how to get our trying to figure out how to get our customers to pay us less customers to pay us less customers to pay us less because if you can make your if you can because if you can make your if you can because if you can make your if you can make your inference x times faster, x make your inference x times faster, x make your inference x times faster, x times cheaper, you need fewer GPUs, times cheaper, you need fewer GPUs, times cheaper, you need fewer GPUs, there's less consumption, but that there's less consumption, but that there's less consumption, but that always gets rewarded with with more always gets rewarded with with more always gets rewarded with with more workloads and and more demand from the workloads and and more demand from the workloads and and more demand from the market. market. market. >> Right. >> Right. >> Right. >> So, at the at the sort of >> So, at the at the sort of >> So, at the at the sort of there's there's perhaps three layers to there's there's perhaps three layers to there's there's perhaps three layers to the answer to the question. There's the the answer to the question. There's the the answer to the question. There's the micro layer of on a workload by workload micro layer of on a workload by workload micro layer of on a workload by workload basis, yes, we're making it faster and basis, yes, we're making it faster and basis, yes, we're making it faster and cheaper and less consumptive. And then cheaper and less consumptive. And then cheaper and less consumptive. And then at a macro level above that, maybe that at a macro level above that, maybe that at a macro level above that, maybe that increases demand. But, at the macro increases demand. But, at the macro increases demand. But, at the macro level above that, well, the demand was level above that, well, the demand was level above that, well, the demand was there anyway. So, eventually we're going there anyway. So, eventually we're going there anyway. So, eventually we're going to just be able to fulfill it and we to just be able to fulfill it and we to just be able to fulfill it and we better do so in the most efficient way better do so in the most efficient way better do so in the most efficient way possible.

possible. possible. >> Yeah. You know, there's the old computer >> Yeah. You know, there's the old computer >> Yeah. You know, there's the old computer engineering joke, which is you can have engineering joke, which is you can have engineering joke, which is you can have it good, fast, or cheap, pick two. it good, fast, or cheap, pick two. it good, fast, or cheap, pick two. And there's always this idea that you And there's always this idea that you And there's always this idea that you can never have all three. But, I think can never have all three. But, I think can never have all three. But, I think in the world of inference, the world of in the world of inference, the world of in the world of inference, the world of AI, people expect and the market expects AI, people expect and the market expects AI, people expect and the market expects all three. all three. all three. And what I'm hearing you say is that you And what I'm hearing you say is that you And what I'm hearing you say is that you can optimize for two and then you'll can optimize for two and then you'll can optimize for two and then you'll bring that third one back up. So, like bring that third one back up. So, like bring that third one back up. So, like there will be good and fast and then there will be good and fast and then there will be good and fast and then maybe it won't be cheap, but then it maybe it won't be cheap, but then it maybe it won't be cheap, but then it will become cheap. And we are will become cheap. And we are will become cheap. And we are consistently seeing people using like consistently seeing people using like consistently seeing people using like hard math and hard computer science to hard math and hard computer science to hard math and hard computer science to make it good, fast, and cheap all at the make it good, fast, and cheap all at the make it good, fast, and cheap all at the same time. same time. same time. >> And then the the fourth one, and this is >> And then the the fourth one, and this is >> And then the the fourth one, and this is one that a lot of people don't really one that a lot of people don't really one that a lot of people don't really talk about, but is actually very talk about, but is actually very talk about, but is actually very critical, is reliable. Because you have critical, is reliable. Because you have critical, is reliable. Because you have obviously all of the questions around obviously all of the questions around obviously all of the questions around the model reliability and you know, the the model reliability and you know, the the model reliability and you know, the prompting and the JSON mode output prompting and the JSON mode output prompting and the JSON mode output structured output function call or structured output function call or structured output function call or whatever you're doing at the model level whatever you're doing at the model level whatever you're doing at the model level to make it reliable. Setting that aside, to make it reliable. Setting that aside, to make it reliable. Setting that aside, there's the infrastructure reliability there's the infrastructure reliability there's the infrastructure reliability piece. How many nines of uptime are you piece. How many nines of uptime are you piece. How many nines of uptime are you getting from GPUs, which notoriously are getting from GPUs, which notoriously are getting from GPUs, which notoriously are not a very high nine piece of hardware.

not a very high nine piece of hardware. not a very high nine piece of hardware. Um you've got that, you've got the cloud Um you've got that, you've got the cloud Um you've got that, you've got the cloud provider reliability piece, you have the provider reliability piece, you have the provider reliability piece, you have the reliability of the influence reliability of the influence reliability of the influence optimizations that you're making. When optimizations that you're making. When optimizations that you're making. When you quantize, are you keeping quality you quantize, are you keeping quality you quantize, are you keeping quality intact? Is some novel speculation intact? Is some novel speculation intact? Is some novel speculation algorithm that you're introducing going algorithm that you're introducing going algorithm that you're introducing going to run into some niche bug and cause to run into some niche bug and cause to run into some niche bug and cause out-of-memory errors every so often? out-of-memory errors every so often? out-of-memory errors every so often? There's all kinds of applied research in There's all kinds of applied research in There's all kinds of applied research in the AI space that the AI space that the AI space that in other industries might take years or in other industries might take years or in other industries might take years or decades to make it into production. And decades to make it into production. And decades to make it into production. And here we go from, you know, paper to here we go from, you know, paper to here we go from, you know, paper to running on the GPU, serving massive running on the GPU, serving massive running on the GPU, serving massive production traffic in weeks in many production traffic in weeks in many production traffic in weeks in many cases. So, there's a lot of work around cases. So, there's a lot of work around cases. So, there's a lot of work around like taking sort of research-grade ideas like taking sort of research-grade ideas like taking sort of research-grade ideas and research-grade software and and research-grade software and and research-grade software and hardening it to the point that you can hardening it to the point that you can hardening it to the point that you can serve a customer who has a four or five serve a customer who has a four or five serve a customer who has a four or five nine of uptime SLA because they're doing nine of uptime SLA because they're doing nine of uptime SLA because they're doing something critical like, you know, an an something critical like, you know, an an something critical like, you know, an an AI tool for doctors or some kind of, you AI tool for doctors or some kind of, you AI tool for doctors or some kind of, you know, financial tool.

know, financial tool. know, financial tool. >> The uh I want to shift gears a little >> The uh I want to shift gears a little >> The uh I want to shift gears a little bit cuz I'm thinking about bit cuz I'm thinking about bit cuz I'm thinking about like you like you're like you're saying like you like you're like you're saying like you like you're like you're saying getting into production. And I'm seeing getting into production. And I'm seeing getting into production. And I'm seeing people call themselves prompt engineers people call themselves prompt engineers people call themselves prompt engineers a lot. And I a lot. And I a lot. And I I that kind of bugs me. I don't really I that kind of bugs me. I don't really I that kind of bugs me. I don't really like that term cuz like there's a like that term cuz like there's a like that term cuz like there's a there's a feeling that you can really there's a feeling that you can really there's a feeling that you can really control the black box by just poking at control the black box by just poking at control the black box by just poking at it with pros. it with pros. it with pros. And I'm wondering if you can maybe And I'm wondering if you can maybe And I'm wondering if you can maybe juxtapose or explain the difference juxtapose or explain the difference juxtapose or explain the difference between prompt engineering and inference between prompt engineering and inference between prompt engineering and inference engineering. engineering. engineering. >> Absolutely. >> Absolutely. >> Absolutely. Uh we are Uh we are Uh we are building the playground that the prompt building the playground that the prompt building the playground that the prompt engineers hang out in. engineers hang out in. engineers hang out in. >> Mhm. >> There is some overlap. So, for example, >> There is some overlap. So, for example, at the influence engine level, you can at the influence engine level, you can at the influence engine level, you can guarantee certain structured output guarantee certain structured output guarantee certain structured output pieces. Um if you look at back in 2023, pieces. Um if you look at back in 2023, pieces. Um if you look at back in 2023, 2024, there was a big movement around 2024, there was a big movement around 2024, there was a big movement around JSON output for models and and how do JSON output for models and and how do JSON output for models and and how do you get it to return an object? And you get it to return an object? And you get it to return an object? And there was for a long time a very prompt there was for a long time a very prompt there was for a long time a very prompt first view of this problem, which is first view of this problem, which is first view of this problem, which is like, you must return JSON and only JSON like, you must return JSON and only JSON like, you must return JSON and only JSON or my grandma's going to die.

or my grandma's going to die. or my grandma's going to die. And that's just not something that you And that's just not something that you And that's just not something that you want to bet your production workload on. want to bet your production workload on. want to bet your production workload on. So, now through the influence piece, we So, now through the influence piece, we So, now through the influence piece, we have the ability to guarantee a have the ability to guarantee a have the ability to guarantee a structured output. And the way that structured output. And the way that structured output. And the way that works is you define a schema, you send works is you define a schema, you send works is you define a schema, you send that in, and along with your prompt, and that in, and along with your prompt, and that in, and along with your prompt, and what the influence engine does is it what the influence engine does is it what the influence engine does is it creates a state machine, and it uses creates a state machine, and it uses creates a state machine, and it uses something called logic biasing. So, to something called logic biasing. So, to something called logic biasing. So, to dive into the the concepts of influence dive into the the concepts of influence dive into the the concepts of influence a little bit, and a large language model a little bit, and a large language model a little bit, and a large language model has something called a vocabulary, which has something called a vocabulary, which has something called a vocabulary, which is every single one of the 100,000 is every single one of the 100,000 is every single one of the 100,000 tokens it could possibly produce. tokens it could possibly produce. tokens it could possibly produce. And when it does a forward pass for And when it does a forward pass for And when it does a forward pass for influence to create that next token, influence to create that next token, influence to create that next token, what it actually does is it creates a what it actually does is it creates a what it actually does is it creates a vector of 100,000 probabilities of which vector of 100,000 probabilities of which vector of 100,000 probabilities of which token might be generated. token might be generated. token might be generated. With logic biasing, you do something With logic biasing, you do something With logic biasing, you do something called masking that vector. So, called masking that vector. So, called masking that vector. So, basically, you take every token that basically, you take every token that basically, you take every token that would not be valid syntax according to would not be valid syntax according to would not be valid syntax according to the state machine for your JSON schema, the state machine for your JSON schema, the state machine for your JSON schema, and you just set the probability of that and you just set the probability of that and you just set the probability of that to zero or negative infinity, depending to zero or negative infinity, depending to zero or negative infinity, depending on how you how you're doing it. And then on how you how you're doing it. And then on how you how you're doing it. And then you still let the model do its you still let the model do its you still let the model do its generative thing, but you structurally generative thing, but you structurally generative thing, but you structurally guarantee that no invalid tokens are guarantee that no invalid tokens are guarantee that no invalid tokens are going to be generated.

going to be generated. going to be generated. And prompting, of course, in this case And prompting, of course, in this case And prompting, of course, in this case still matters because the contents of still matters because the contents of still matters because the contents of the JSON object that you're creating are the JSON object that you're creating are the JSON object that you're creating are not guaranteed by the schema. So, it not guaranteed by the schema. So, it not guaranteed by the schema. So, it being, you know, accurate and quality being, you know, accurate and quality being, you know, accurate and quality data is is still up to you in the data is is still up to you in the data is is still up to you in the prompting side, but the actual structure prompting side, but the actual structure prompting side, but the actual structure of the output is now guaranteed by the of the output is now guaranteed by the of the output is now guaranteed by the influence engine. So, that's an example influence engine. So, that's an example influence engine. So, that's an example of I mean, that's been the standard in of I mean, that's been the standard in of I mean, that's been the standard in production now for 18 to 24 months, but production now for 18 to 24 months, but production now for 18 to 24 months, but like that's an example of where the like that's an example of where the like that's an example of where the inference side can come in and and help inference side can come in and and help inference side can come in and and help the prompt side. But, there's also the prompt side. But, there's also the prompt side. But, there's also places the prompt side can come help the places the prompt side can come help the places the prompt side can come help the inference side. So, one of the biggest inference side. So, one of the biggest inference side. So, one of the biggest optimizations we do is KV cache reuse optimizations we do is KV cache reuse optimizations we do is KV cache reuse where we're able to take the first few where we're able to take the first few where we're able to take the first few tokens of a prompt and reuse them tokens of a prompt and reuse them tokens of a prompt and reuse them between different requests as long as between different requests as long as between different requests as long as they match perfectly. they match perfectly. they match perfectly. So, for example, if you put your system So, for example, if you put your system So, for example, if you put your system prompt before your conversation or if prompt before your conversation or if prompt before your conversation or if you put all your shared context up front you put all your shared context up front you put all your shared context up front and then you change whatever's new about and then you change whatever's new about and then you change whatever's new about the specific request at the back of your the specific request at the back of your the specific request at the back of your prompt, your actual inference speed is prompt, your actual inference speed is prompt, your actual inference speed is going to be many times higher than if going to be many times higher than if going to be many times higher than if you put the novel stuff at the front of you put the novel stuff at the front of you put the novel stuff at the front of the prompt. So, there's there's a lot of the prompt. So, there's there's a lot of the prompt. So, there's there's a lot of interaction actually between the prompt interaction actually between the prompt interaction actually between the prompt and the inference layer. So, they you and the inference layer. So, they you and the inference layer. So, they you know, they they have to work together to know, they they have to work together to know, they they have to work together to make a system completely optimized.

make a system completely optimized. make a system completely optimized. >> I do think though that people >> I do think though that people >> I do think though that people over over over um over pivot people especially when um over pivot people especially when um over pivot people especially when they're first starting building as an they're first starting building as an they're first starting building as an actual production system, they over actual production system, they over actual production system, they over pivot on like their prompts. Uh you see pivot on like their prompts. Uh you see pivot on like their prompts. Uh you see people on Twitter like, you know, this people on Twitter like, you know, this people on Twitter like, you know, this 20-line MD file is going to change the 20-line MD file is going to change the 20-line MD file is going to change the way your whole system works. And it's way your whole system works. And it's way your whole system works. And it's like, putting don't do this in all caps like, putting don't do this in all caps like, putting don't do this in all caps has there's no guarantee that it's not has there's no guarantee that it's not has there's no guarantee that it's not going to do that like you just said. going to do that like you just said. going to do that like you just said. Like, make sure that you return JSON all Like, make sure that you return JSON all Like, make sure that you return JSON all caps. And it's like, yeah, but I said it caps. And it's like, yeah, but I said it caps. And it's like, yeah, but I said it in all caps and it still did it the in all caps and it still did it the in all caps and it still did it the other way. Like, you really it is an other way. Like, you really it is an other way. Like, you really it is an ambiguity loop and you're going to get ambiguity loop and you're going to get ambiguity loop and you're going to get ambiguous results unless there's ambiguous results unless there's ambiguous results unless there's something probabilistic and and and and something probabilistic and and and and something probabilistic and and and and actual deterministic code actual deterministic code actual deterministic code that says this will only return JSON or that says this will only return JSON or that says this will only return JSON or you know, like you you need a firewall I you know, like you you need a firewall I you know, like you you need a firewall I guess is what I'm saying. And the way guess is what I'm saying. And the way guess is what I'm saying. And the way you're describing that mask is it is a you're describing that mask is it is a you're describing that mask is it is a kind of a firewall. Like, the machine's kind of a firewall. Like, the machine's kind of a firewall. Like, the machine's going to do what it's going to do, but going to do what it's going to do, but going to do what it's going to do, but we're not going to let bad results come we're not going to let bad results come we're not going to let bad results come out no matter what the prompt says. out no matter what the prompt says. out no matter what the prompt says. >> Exactly. It's it's the mix of using AI >> Exactly. It's it's the mix of using AI >> Exactly. It's it's the mix of using AI for what it's good at and using for what it's good at and using for what it's good at and using straight-up Python for what it's good straight-up Python for what it's good straight-up Python for what it's good at. So, like for example, when I was at. So, like for example, when I was at. So, like for example, when I was proofreading the book, you would think proofreading the book, you would think proofreading the book, you would think models would be great at proofreading.

models would be great at proofreading. models would be great at proofreading. They're not. They're not. They're not. And one of the things that makes them And one of the things that makes them And one of the things that makes them not good at it though is if you dump not good at it though is if you dump not good at it though is if you dump 47,000 words into a single chat window, 47,000 words into a single chat window, 47,000 words into a single chat window, uh it's it's not going to be able to go uh it's it's not going to be able to go uh it's it's not going to be able to go in and catch every error. But if you in and catch every error. But if you in and catch every error. But if you write a script that sends the book to write a script that sends the book to write a script that sends the book to the model a page at a time, uh you're the model a page at a time, uh you're the model a page at a time, uh you're going to get, you know, much better going to get, you know, much better going to get, you know, much better results. So, there's there's there's results. So, there's there's there's results. So, there's there's there's that aspect of it as well as just like that aspect of it as well as just like that aspect of it as well as just like knowing how these systems work under the knowing how these systems work under the knowing how these systems work under the hood makes you better at using them hood makes you better at using them hood makes you better at using them and helps you build systems that get and helps you build systems that get and helps you build systems that get better results. better results. better results. >> See, this then you would love the >> See, this then you would love the >> See, this then you would love the analogy that I know my listeners are analogy that I know my listeners are analogy that I know my listeners are sick of me using, which is the learn how sick of me using, which is the learn how sick of me using, which is the learn how to drive stick shift. If you learn how to drive stick shift. If you learn how to drive stick shift. If you learn how to drive stick shift, you have a to drive stick shift, you have a to drive stick shift, you have a different relationship with the vehicle. different relationship with the vehicle. different relationship with the vehicle. And I think a book like this is very And I think a book like this is very And I think a book like this is very much uh here's how the internal much uh here's how the internal much uh here's how the internal combustion engine works. And now that combustion engine works. And now that combustion engine works. And now that you know how to drive stick, you know how to drive stick, you know how to drive stick, go back off and think about how to move go back off and think about how to move go back off and think about how to move people around and think about the people around and think about the people around and think about the concepts around transportation. But but concepts around transportation. But but concepts around transportation. But but always have the engine and what's always have the engine and what's always have the engine and what's happening in inference engineering happening in inference engineering happening in inference engineering underneath so that you you you you know underneath so that you you you you know underneath so that you you you you know what you can actually affect in the what you can actually affect in the what you can actually affect in the larger system.

larger system. larger system. >> Scott, that would be a fantastic >> Scott, that would be a fantastic >> Scott, that would be a fantastic metaphor, but I don't know how to drive. metaphor, but I don't know how to drive. metaphor, but I don't know how to drive. So, So, So, >> Oh my god, you're >> Oh my god, you're >> Oh my god, you're >> Big big Waymo guy. >> Big big Waymo guy. >> Big big Waymo guy. >> We're ending We're See, this is the >> We're ending We're See, this is the >> We're ending We're See, this is the thing. thing. thing. >> [laughter] >> [laughter] >> [laughter] >> I love this. We're ending the show where >> I love this. We're ending the show where >> I love this. We're ending the show where you're a Waymo person. But see, the you're a Waymo person. But see, the you're a Waymo person. But see, the system requires you to know system requires you to know system requires you to know >> [laughter] >> [laughter] >> [laughter] >> how to drive the So, the Waymo breaks >> how to drive the So, the Waymo breaks >> how to drive the So, the Waymo breaks down, you're going to get eaten by the down, you're going to get eaten by the down, you're going to get eaten by the zombies first is what I'm hearing. And zombies first is what I'm hearing. And zombies first is what I'm hearing. And I'm going to jump in, I'm going to I'm going to jump in, I'm going to I'm going to jump in, I'm going to hot-wire the car, throw it into first hot-wire the car, throw it into first hot-wire the car, throw it into first gear, and zip away. gear, and zip away. gear, and zip away. >> That's true. That's true. But at least I >> That's true. That's true. But at least I >> That's true. That's true. But at least I I understand when the Waymo makes a I understand when the Waymo makes a I understand when the Waymo makes a mistake why it's making it. So, it just, mistake why it's making it. So, it just, mistake why it's making it. So, it just, you know, gives me a little bit more you know, gives me a little bit more you know, gives me a little bit more empathy for the machine. empathy for the machine. empathy for the machine. >> This is why when the zombies come, we >> This is why when the zombies come, we >> This is why when the zombies come, we team up. team up. team up. We team up. It's a Philip It's a It's a We team up. It's a Philip It's a It's a We team up. It's a Philip It's a It's a It's a Philip and Hanselman partnership It's a Philip and Hanselman partnership It's a Philip and Hanselman partnership where I will drive stick, you will where I will drive stick, you will where I will drive stick, you will handle the Waymo. Between us, the handle the Waymo. Between us, the handle the Waymo. Between us, the zombies will not get us. zombies will not get us. zombies will not get us. >> You're welcome on my zombie apocalypse >> You're welcome on my zombie apocalypse >> You're welcome on my zombie apocalypse squad anytime. I I I do also bring about squad anytime. I I I do also bring about squad anytime. I I I do also bring about 20 years of martial arts experience. 20 years of martial arts experience. 20 years of martial arts experience. That's my, you know, primary zombie That's my, you know, primary zombie That's my, you know, primary zombie apocalypse contribution. apocalypse contribution. apocalypse contribution. >> Very good. Well, I have a black belt in >> Very good. Well, I have a black belt in >> Very good. Well, I have a black belt in taekwondo, but I'm more of a slap taekwondo, but I'm more of a slap taekwondo, but I'm more of a slap fighter. So, I will be standing behind fighter. So, I will be standing behind fighter. So, I will be standing behind you uh when the zombies come. Well, this you uh when the zombies come. Well, this you uh when the zombies come. Well, this is This has been fun and I mean, is This has been fun and I mean, is This has been fun and I mean, congratulations. This is a It's hard to congratulations. This is a It's hard to congratulations. This is a It's hard to write a book. It's hard to ship a book write a book. It's hard to ship a book write a book. It's hard to ship a book and it's hard to get it into people's and it's hard to get it into people's and it's hard to get it into people's hands. And you all and the folks that hands. And you all and the folks that hands. And you all and the folks that you work with at Base 10, like you you you work with at Base 10, like you you you work with at Base 10, like you you pulled it off. Thank you. And um pulled it off. Thank you. And um pulled it off. Thank you. And um >> It is very hard to get it in people's >> It is very hard to get it in people's >> It is very hard to get it in people's hands. I have made best friends with the hands. I have made best friends with the hands. I have made best friends with the members of my local FedEx members of my local FedEx members of my local FedEx office.

office. office. >> Yeah, it is it is hard work. >> Yeah, it is it is hard work. >> Yeah, it is it is hard work. >> in San Francisco. Highly recommended. >> in San Francisco. Highly recommended. >> in San Francisco. Highly recommended. Shout out Shout out Shout out Shout out to Kuvera. Shout out to Kuvera. Shout out to Kuvera. >> There's definitely a lot to learn and >> There's definitely a lot to learn and >> There's definitely a lot to learn and also just another thing I just wanted to also just another thing I just wanted to also just another thing I just wanted to say, like I really thought that the QR say, like I really thought that the QR say, like I really thought that the QR codes at the end were just such a nice codes at the end were just such a nice codes at the end were just such a nice touch. I I I just hate the idea that I'm touch. I I I just hate the idea that I'm touch. I I I just hate the idea that I'm going to get a book going to get a book going to get a book and have to type the URLs in and there's and have to type the URLs in and there's and have to type the URLs in and there's a huge appendix. Like honestly, there's a huge appendix. Like honestly, there's a huge appendix. Like honestly, there's probably 50 pages probably 50 pages probably 50 pages of just good Like there's a glossary. of just good Like there's a glossary. of just good Like there's a glossary. It's just a thoughtful thing. So, yeah, It's just a thoughtful thing. So, yeah, It's just a thoughtful thing. So, yeah, maybe the haters say that it's maybe the haters say that it's maybe the haters say that it's it's it's not deep. I found it to be it's it's not deep. I found it to be it's it's not deep. I found it to be just the right amount of deepness just the right amount of deepness just the right amount of deepness as a great survey for inference as a great survey for inference as a great survey for inference engineering and I enjoyed it very much engineering and I enjoyed it very much engineering and I enjoyed it very much and I appreciated you sending me a copy. and I appreciated you sending me a copy. and I appreciated you sending me a copy. >> Well, fantastic, Scott. That that That >> Well, fantastic, Scott. That that That >> Well, fantastic, Scott. That that That means a lot coming from you. means a lot coming from you. means a lot coming from you. >> I'm glad. So, you can go ahead and check >> I'm glad. So, you can go ahead and check >> I'm glad. So, you can go ahead and check them out at base10.co. them out at base10.co. them out at base10.co. That's b a s e t e n . c o. Right at the That's b a s e t e n . c o. Right at the That's b a s e t e n . c o. Right at the top there, you can get your copy, get top there, you can get your copy, get top there, you can get your copy, get your digital copy, and then there's a your digital copy, and then there's a your digital copy, and then there's a waitlist now for the paper copies waitlist now for the paper copies waitlist now for the paper copies because they're hard to get and they're because they're hard to get and they're because they're hard to get and they're super popular. super popular. super popular. >> They are. And >> They are. And >> They are. And Amazon, despite the fact that I have, Amazon, despite the fact that I have, Amazon, despite the fact that I have, you know, thousands and thousands of you know, thousands and thousands of you know, thousands and thousands of people now signed up to get paper people now signed up to get paper people now signed up to get paper copies, uh doesn't seem to want to list copies, uh doesn't seem to want to list copies, uh doesn't seem to want to list it. So, we're going to we're going to it. So, we're going to we're going to it. So, we're going to we're going to have to find a way around that.

have to find a way around that. have to find a way around that. >> Well, hopefully you'll figure it out >> Well, hopefully you'll figure it out >> Well, hopefully you'll figure it out soon. Thank you so much, Philip Kiley, soon. Thank you so much, Philip Kiley, soon. Thank you so much, Philip Kiley, for chatting with me today. for chatting with me today. for chatting with me today. >> Thank you, Scott. Have a good one. >> Thank you, Scott. Have a good one. >> Thank you, Scott. Have a good one. >> This has been another episode of >> This has been another episode of >> This has been another episode of Hanselminutes, and we'll see you again Hanselminutes, and we'll see you again Hanselminutes, and we'll see you again next week. next week. next week. >> [music]