Inference Engineering with Baseten's Philip Kiely Inference Engineering with Baseten's Philip Kiely

Did Did you feel uncomfortable that you Did Did you feel uncomfortable that you don't have a PhD in in machine learning don't have a PhD in in machine learning don't have a PhD in in machine learning and deep learning? Did you Did you have and deep learning? Did you Did you have and deep learning? Did you Did you have Did you Did you say like, "Should I be Did you Did you say like, "Should I be Did you Did you say like, "Should I be the one that writes the book?" How did the one that writes the book?" How did the one that writes the book?" How did you deal with that? Cuz I always feel you deal with that? Cuz I always feel you deal with that? Cuz I always feel like an impostor when I'm writing like an impostor when I'm writing like an impostor when I'm writing technical books. technical books. technical books. I am blessed with a complete absence of I am blessed with a complete absence of I am blessed with a complete absence of impostor syndrome. Oh, wow. You're the impostor syndrome. Oh, wow. You're the impostor syndrome. Oh, wow. You're the first. That's amazing. Tell us your first. That's amazing. Tell us your first. That's amazing. Tell us your ways. Perhaps some people in my life ways. Perhaps some people in my life ways. Perhaps some people in my life would would would tell me that that a would would would tell me that that a would would would tell me that that a little bit of it might be good for me, little bit of it might be good for me, little bit of it might be good for me, but Hey friends, you probably knew that but Hey friends, you probably knew that but Hey friends, you probably knew that TextControl is a powerful library for TextControl is a powerful library for TextControl is a powerful library for document editing and PDF generation, but document editing and PDF generation, but document editing and PDF generation, but did you also know that they're a strong did you also know that they're a strong did you also know that they're a strong supporter of the developer community? supporter of the developer community? supporter of the developer community? And it's part of their mission to build And it's part of their mission to build And it's part of their mission to build and support a strong community by being and support a strong community by being and support a strong community by being present, by listening to users, and by present, by listening to users, and by present, by listening to users, and by sharing knowledge at conferences across sharing knowledge at conferences across sharing knowledge at conferences across Europe and the United States. If you're Europe and the United States. If you're Europe and the United States. If you're heading to a conference soon, heading to a conference soon, heading to a conference soon, maybe check if TextControl will be maybe check if TextControl will be maybe check if TextControl will be there. Stop by and say hi. You'll find there. Stop by and say hi. You'll find there. Stop by and say hi. You'll find their full conference calendar at their full conference calendar at their full conference calendar at textcontrol.com. textcontrol.com. textcontrol.com. That's t e x t control.com.

Hey friends, I'm Scott Hanselman, and Hey friends, I'm Scott Hanselman, and this is another episode of this is another episode of this is another episode of Hanselminutes. And today I'm chatting Hanselminutes. And today I'm chatting Hanselminutes. And today I'm chatting with Philip Caley. He is the author of with Philip Caley. He is the author of with Philip Caley. He is the author of Inference Engineering, and he works at Inference Engineering, and he works at Inference Engineering, and he works at Basecamp. You can check it out at Basecamp. You can check it out at Basecamp. You can check it out at basecamp.co. basecamp.co. basecamp.co. How's it going, sir? I'm doing great, How's it going, sir? I'm doing great, How's it going, sir? I'm doing great, Scott. It's an honor to be here. Well, Scott. It's an honor to be here. Well, Scott. It's an honor to be here. Well, it's cool to hang out with you. So, it's cool to hang out with you. So, it's cool to hang out with you. So, this book came out of nowhere, and you this book came out of nowhere, and you this book came out of nowhere, and you you you tweeted about it. I saw it. I you you tweeted about it. I saw it. I you you tweeted about it. I saw it. I saw the opening video, and then saw the opening video, and then saw the opening video, and then you were kind enough to send me a copy, you were kind enough to send me a copy, you were kind enough to send me a copy, which I have read and dog-eared, and which I have read and dog-eared, and which I have read and dog-eared, and I've got a bunch of dog-eared pages here I've got a bunch of dog-eared pages here I've got a bunch of dog-eared pages here with questions that I want to ask you with questions that I want to ask you with questions that I want to ask you about. But I I got to just shout out about. But I I got to just shout out about. But I I got to just shout out like like like I There's just so much kind of like slop I There's just so much kind of like slop I There's just so much kind of like slop right now. There's so much AI slop. right now. There's so much AI slop. right now. There's so much AI slop. And then a book shows up that kind of And then a book shows up that kind of And then a book shows up that kind of like is the opposite in the sense of like is the opposite in the sense of like is the opposite in the sense of like it smells good. It's got a really like it smells good. It's got a really like it smells good. It's got a really nice design. Like Like I was like the nice design. Like Like I was like the nice design. Like Like I was like the first thing I thought was like, "I hope first thing I thought was like, "I hope first thing I thought was like, "I hope they didn't AI generate the design." And they didn't AI generate the design." And they didn't AI generate the design." And then you actually shout out the author then you actually shout out the author then you actually shout out the author in the book who did the design custom. in the book who did the design custom. in the book who did the design custom. Like it feels bespoke in a moment where Like it feels bespoke in a moment where Like it feels bespoke in a moment where everything's not bespoke. Was that a everything's not bespoke. Was that a everything's not bespoke. Was that a conscious decision to do something like conscious decision to do something like conscious decision to do something like that?

that? that? Absolutely. I mean, the thing is, Scott, Absolutely. I mean, the thing is, Scott, Absolutely. I mean, the thing is, Scott, I love writing books. I've [snorts] done I love writing books. I've [snorts] done I love writing books. I've [snorts] done it I've done it before, but I've never it I've done it before, but I've never it I've done it before, but I've never done a good job like this. So, it took done a good job like this. So, it took done a good job like this. So, it took me a few tries to to get here. So, I me a few tries to to get here. So, I me a few tries to to get here. So, I wrote I wrote Influence Engineering wrote I wrote Influence Engineering wrote I wrote Influence Engineering because number one, I love the topic, because number one, I love the topic, because number one, I love the topic, and number two, I love writing books. and number two, I love writing books. and number two, I love writing books. And if I was, you know, just saying I And if I was, you know, just saying I And if I was, you know, just saying I mean, you saw in the launch video, the mean, you saw in the launch video, the mean, you saw in the launch video, the the very first line of the video was, the very first line of the video was, the very first line of the video was, "Hey, you know, they made me take it "Hey, you know, they made me take it "Hey, you know, they made me take it out, but I was going to say, 'Hey, out, but I was going to say, 'Hey, out, but I was going to say, 'Hey, Claude, write me a book.'" I just said, Claude, write me a book.'" I just said, Claude, write me a book.'" I just said, "Hey, hey, write me a book "Hey, hey, write me a book "Hey, hey, write me a book called Influence Engineering. Make no called Influence Engineering. Make no called Influence Engineering. Make no mistakes." It it it it just doesn't mistakes." It it it it just doesn't mistakes." It it it it just doesn't work. You just can't do it. So, Well, work. You just can't do it. So, Well, work. You just can't do it. So, Well, that's fair. that's fair. that's fair. >> write something myself. We got to We We >> write something myself. We got to We We >> write something myself. We got to We We got to say right off the bat, when got to say right off the bat, when got to say right off the bat, when you're writing a book about AI, you're you're writing a book about AI, you're you're writing a book about AI, you're writing a book about the technologies writing a book about the technologies writing a book about the technologies that power AI, the first question that that power AI, the first question that that power AI, the first question that an interviewer has to ask is, an interviewer has to ask is, an interviewer has to ask is, did you AI slop the book and and have did you AI slop the book and and have did you AI slop the book and and have the machine write the book for you, or the machine write the book for you, or the machine write the book for you, or is this 4 years of actually writing is this 4 years of actually writing is this 4 years of actually writing hard? hard? hard? So, I tried my best to AI slop this So, I tried my best to AI slop this So, I tried my best to AI slop this because I I came up with the idea for because I I came up with the idea for because I I came up with the idea for this book 6 months ago. And, you know, this book 6 months ago. And, you know, this book 6 months ago. And, you know, if you go talk to a traditional if you go talk to a traditional if you go talk to a traditional publisher, which I did, they'll tell publisher, which I did, they'll tell publisher, which I did, they'll tell you, "Once you give me a manuscript, you, "Once you give me a manuscript, you, "Once you give me a manuscript, it'll take 12 to 18 months to get this it'll take 12 to 18 months to get this it'll take 12 to 18 months to get this thing out into the world." And I knew thing out into the world." And I knew thing out into the world." And I knew that I wanted to go from idea to that I wanted to go from idea to that I wanted to go from idea to finished hardcovers all over the world finished hardcovers all over the world finished hardcovers all over the world in 6 months. And so, anything that I in 6 months. And so, anything that I in 6 months. And so, anything that I could do to make that faster, I wanted could do to make that faster, I wanted could do to make that faster, I wanted to do. So, 100% I tried. I said, "Hey,

to do. So, 100% I tried. I said, "Hey, to do. So, 100% I tried. I said, "Hey, you know, hey, can can you come up with you know, hey, can can you come up with you know, hey, can can you come up with the outline? Can you write this chapter, the outline? Can you write this chapter, the outline? Can you write this chapter, write that chapter?" But the quality was write that chapter?" But the quality was write that chapter?" But the quality was just completely unusable. There were a just completely unusable. There were a just completely unusable. There were a few places where I was able to use AI. few places where I was able to use AI. few places where I was able to use AI. So, number one, So, number one, So, number one, by the way, Scott, like have you done by the way, Scott, like have you done by the way, Scott, like have you done any long-form writing? I assume I assume any long-form writing? I assume I assume any long-form writing? I assume I assume you have in in your career. you have in in your career. you have in in your career. >> I've written a couple of books. I've got >> I've written a couple of books. I've got >> I've written a couple of books. I've got a And And none of them used AI, and some a And And none of them used AI, and some a And And none of them used AI, and some of them are almost a thousand pages of them are almost a thousand pages of them are almost a thousand pages long, and it's mostly you know, it's long, and it's mostly you know, it's long, and it's mostly you know, it's code samples. Like code samples. Like code samples. Like I I mean, like this is the thing. Like I I mean, like this is the thing. Like I I mean, like this is the thing. Like ironically, we're talking about a book ironically, we're talking about a book ironically, we're talking about a book about AI. It doesn't feel like there's about AI. It doesn't feel like there's about AI. It doesn't feel like there's any AI. I I read the book. I I got the any AI. I I read the book. I I got the any AI. I I read the book. I I got the book marked up. None of my books are book marked up. None of my books are book marked up. None of my books are used AI because it's just it doesn't used AI because it's just it doesn't used AI because it's just it doesn't hold a throughline. It doesn't hold a hold a throughline. It doesn't hold a hold a throughline. It doesn't hold a topic long enough. Exactly. So, let me topic long enough. Exactly. So, let me topic long enough. Exactly. So, let me tell you how to use AI if you're writing tell you how to use AI if you're writing tell you how to use AI if you're writing a book because, yeah, for the most part a book because, yeah, for the most part a book because, yeah, for the most part for the most part I didn't. If you look for the most part I didn't. If you look for the most part I didn't. If you look at, for example, chapter four, at, for example, chapter four, at, for example, chapter four, section the figure 4.1, this guy right section the figure 4.1, this guy right section the figure 4.1, this guy right here. It's a six-line CUDA kernel. here. It's a six-line CUDA kernel. here. It's a six-line CUDA kernel. Okay.

Okay. Okay. Got it. I asked for an example CUDA Got it. I asked for an example CUDA Got it. I asked for an example CUDA kernel. That saved me 30 minutes, you kernel. That saved me 30 minutes, you kernel. That saved me 30 minutes, you know. know. know. >> I see. So, the code So, some of the code >> I see. So, the code So, some of the code >> I see. So, the code So, some of the code might be. You know, you can it's it's might be. You know, you can it's it's might be. You know, you can it's it's small it's small pieces like that that small it's small pieces like that that small it's small pieces like that that saved me 30 minutes there, an hour there saved me 30 minutes there, an hour there saved me 30 minutes there, an hour there that you can verify yourself and and that you can verify yourself and and that you can verify yourself and and make sure they're good. It's, you know, make sure they're good. It's, you know, make sure they're good. It's, you know, like I said, you you've written before. like I said, you you've written before. like I said, you you've written before. how hard it is to look at the blank how hard it is to look at the blank how hard it is to look at the blank page. It's coming up with a few bad page. It's coming up with a few bad page. It's coming up with a few bad ideas so that I can throw those away and ideas so that I can throw those away and ideas so that I can throw those away and be inspired to create a good one. It's be inspired to create a good one. It's be inspired to create a good one. It's taking your outline or taking your taking your outline or taking your taking your outline or taking your sections and checking them for factual sections and checking them for factual sections and checking them for factual accuracy, checking them for for accuracy, checking them for for accuracy, checking them for for completeness, but you can't just take completeness, but you can't just take completeness, but you can't just take the outputs and say, "Oh, well, every the outputs and say, "Oh, well, every the outputs and say, "Oh, well, every fact check it just gave me means all of fact check it just gave me means all of fact check it just gave me means all of this is wrong." No, that's just your this is wrong." No, that's just your this is wrong." No, that's just your to-do list to go do manually yourself. to-do list to go do manually yourself. to-do list to go do manually yourself. And then if you look in the back, like And then if you look in the back, like And then if you look in the back, like every time I write a book, I write every time I write a book, I write every time I write a book, I write custom software to make writing and custom software to make writing and custom software to make writing and laying out the book easier. So, for laying out the book easier. So, for laying out the book easier. So, for example, like these QR codes and the example, like these QR codes and the example, like these QR codes and the layout of that. So, layout of that. So, layout of that. So, I didn't, you know, AI generate that I didn't, you know, AI generate that I didn't, you know, AI generate that because you can't. It's not going to be because you can't. It's not going to be because you can't. It's not going to be good, but I had Koso write me a script good, but I had Koso write me a script good, but I had Koso write me a script to do that. In appendix A, it's to do that. In appendix A, it's to do that. In appendix A, it's alphabetized, you know. I actually tried alphabetized, you know. I actually tried alphabetized, you know. I actually tried pasting the entirety of appendix A into pasting the entirety of appendix A into pasting the entirety of appendix A into ChatGPT, and I said, "Hey, alphabetize ChatGPT, and I said, "Hey, alphabetize ChatGPT, and I said, "Hey, alphabetize this." And it it got it wrong. And then this." And it it got it wrong. And then this." And it it got it wrong. And then I said, "Hey, Koso, write me a script I said, "Hey, Koso, write me a script I said, "Hey, Koso, write me a script that will alphabetize this." And it made that will alphabetize this." And it made that will alphabetize this." And it made 20 lines of Python, and it alphabetized 20 lines of Python, and it alphabetized 20 lines of Python, and it alphabetized it perfectly. So, the the main way that

it perfectly. So, the the main way that it perfectly. So, the the main way that I used AI to write this book was not the I used AI to write this book was not the I used AI to write this book was not the actual words, was not the diagrams. The actual words, was not the diagrams. The actual words, was not the diagrams. The words are by me. The diagrams are by words are by me. The diagrams are by words are by me. The diagrams are by human artists and human designers. It's human artists and human designers. It's human artists and human designers. It's to write the code that makes everything to write the code that makes everything to write the code that makes everything else work faster in the writing, else work faster in the writing, else work faster in the writing, editing, and publishing process. editing, and publishing process. editing, and publishing process. And see, and I feel like, you know, in a And see, and I feel like, you know, in a And see, and I feel like, you know, in a moment where there's just so much crap moment where there's just so much crap moment where there's just so much crap being generated, I think that is a good being generated, I think that is a good being generated, I think that is a good example of how you want to use AI. I example of how you want to use AI. I example of how you want to use AI. I always use the word toil. If it's always use the word toil. If it's always use the word toil. If it's tedious and it's toil, then AI will do tedious and it's toil, then AI will do tedious and it's toil, then AI will do it. Like you say, generating QR codes or it. Like you say, generating QR codes or it. Like you say, generating QR codes or writing scripts to to to alphabetize writing scripts to to to alphabetize writing scripts to to to alphabetize things. But if it's something like like things. But if it's something like like things. But if it's something like like the like the cover. Like I immediately the like the cover. Like I immediately the like the cover. Like I immediately looked at the cover, and I was like, looked at the cover, and I was like, looked at the cover, and I was like, "This is really cool." And I wanted to "This is really cool." And I wanted to "This is really cool." And I wanted to see the human name of the person that see the human name of the person that see the human name of the person that wrote the cover. You know what I mean? wrote the cover. You know what I mean? wrote the cover. You know what I mean? And I like this you know, it's like And I like this you know, it's like And I like this you know, it's like cover by Luke de Haas, you know? And cover by Luke de Haas, you know? And cover by Luke de Haas, you know? And shout out to Luke. And I don't know. shout out to Luke. And I don't know. shout out to Luke. And I don't know. It's just like I want people want both. It's just like I want people want both. It's just like I want people want both. And you want the hard parts that humans And you want the hard parts that humans And you want the hard parts that humans work on to to feel hard, and I can feel work on to to feel hard, and I can feel work on to to feel hard, and I can feel that you were probably up late that you were probably up late that you were probably up late working [laughter] on this book. Are you working [laughter] on this book. Are you working [laughter] on this book. Are you okay? Did you hurt yourself in the okay? Did you hurt yourself in the okay? Did you hurt yourself in the writing of the book? There were There writing of the book? There were There writing of the book? There were There were a lot of very long writing days.

were a lot of very long writing days. were a lot of very long writing days. The The actual drafting piece was about The The actual drafting piece was about The The actual drafting piece was about a 6-week sprint where 4 days a week I a 6-week sprint where 4 days a week I a 6-week sprint where 4 days a week I would wake up at 5:00 in the morning and would wake up at 5:00 in the morning and would wake up at 5:00 in the morning and write until I fell asleep, and then the write until I fell asleep, and then the write until I fell asleep, and then the other 3 days I'd go to work and do my other 3 days I'd go to work and do my other 3 days I'd go to work and do my actual job. actual job. actual job. So, yeah, and and of course, shout out So, yeah, and and of course, shout out So, yeah, and and of course, shout out to Luke. He's the one of the brand to Luke. He's the one of the brand to Luke. He's the one of the brand designers that we have at Basecamp, and designers that we have at Basecamp, and designers that we have at Basecamp, and and he did a fantastic job on the cover and he did a fantastic job on the cover and he did a fantastic job on the cover and setting the design language for the and setting the design language for the and setting the design language for the interior. There's in chapter one this interior. There's in chapter one this interior. There's in chapter one this little diagram of some football players. little diagram of some football players. little diagram of some football players. That's my favorite art in the entire That's my favorite art in the entire That's my favorite art in the entire book. book. book. Yeah, it it was I think this project was Yeah, it it was I think this project was Yeah, it it was I think this project was a great example of collaboration between a great example of collaboration between a great example of collaboration between [clears throat] multiple humans and [clears throat] multiple humans and [clears throat] multiple humans and between AI tools because writing a book between AI tools because writing a book between AI tools because writing a book is toil. This to overuse a metaphor, is toil. This to overuse a metaphor, is toil. This to overuse a metaphor, it's like running a marathon. No it's like running a marathon. No it's like running a marathon. No individual step is that hard. The hard individual step is that hard. The hard individual step is that hard. The hard part is taking all the steps part is taking all the steps part is taking all the steps back-to-back. Not every piece of toil back-to-back. Not every piece of toil back-to-back. Not every piece of toil can be can be automated. For example, can be can be automated. For example, can be can be automated. For example, that book you're holding in your hand is that book you're holding in your hand is that book you're holding in your hand is signed. Do you know how long it takes to signed. Do you know how long it takes to signed. Do you know how long it takes to sign 300 hardcovers? Cuz I just found sign 300 hardcovers? Cuz I just found sign 300 hardcovers? Cuz I just found out. I I did note your hands were a out. I I did note your hands were a out. I I did note your hands were a little shaky when you when you when you little shaky when you when you when you little shaky when you when you when you signed when you [laughter] signed the signed when you [laughter] signed the signed when you [laughter] signed the book.

book. book. >> My handwriting's just that bad, Scott. >> My handwriting's just that bad, Scott. >> My handwriting's just that bad, Scott. That's just how it is. That's just how it is. That's just how it is. The other thing that I think is worth The other thing that I think is worth The other thing that I think is worth noting, and I want to shout out like noting, and I want to shout out like noting, and I want to shout out like people who might buy the book or might people who might buy the book or might people who might buy the book or might think about getting into the digital think about getting into the digital think about getting into the digital download or a paper copy is that this is download or a paper copy is that this is download or a paper copy is that this is a dense book. There's a lot of garbage a dense book. There's a lot of garbage a dense book. There's a lot of garbage out there that is just stuff you can out there that is just stuff you can out there that is just stuff you can learn online. learn online. learn online. What I appreciated about your effort was What I appreciated about your effort was What I appreciated about your effort was that I had to work pretty hard that I had to work pretty hard that I had to work pretty hard at this book. Like I had to at this book. Like I had to at this book. Like I had to at points I had to stop, look stuff up. at points I had to stop, look stuff up. at points I had to stop, look stuff up. There were a couple of times where I did There were a couple of times where I did There were a couple of times where I did ChatGPT voice mode and asked for to ChatGPT voice mode and asked for to ChatGPT voice mode and asked for to understand stuff better. It is It is understand stuff better. It is It is understand stuff better. It is It is dense, and it seems surprising to me dense, and it seems surprising to me dense, and it seems surprising to me that a company that a company that a company that makes AI easier would write a book that makes AI easier would write a book that makes AI easier would write a book about the hard stuff. How did you decide about the hard stuff. How did you decide about the hard stuff. How did you decide to write a hard book rather than to like to write a hard book rather than to like to write a hard book rather than to like make a puff piece? make a puff piece? make a puff piece? Well, for one thing, to me, this is this Well, for one thing, to me, this is this Well, for one thing, to me, this is this is kind of not the hard stuff. The The is kind of not the hard stuff. The The is kind of not the hard stuff. The The hard stuff that the model performance hard stuff that the model performance hard stuff that the model performance engineers are doing every day and the engineers are doing every day and the engineers are doing every day and the infrastructure engineers, you know, I'm infrastructure engineers, you know, I'm infrastructure engineers, you know, I'm just I'm just a friendly dev well with a just I'm just a friendly dev well with a just I'm just a friendly dev well with a with an undergrad CS degree.

with an undergrad CS degree. with an undergrad CS degree. No, I I I get what you mean. That's No, I I I get what you mean. That's No, I I I get what you mean. That's another reason that I couldn't use very another reason that I couldn't use very another reason that I couldn't use very much AI to write this. It's like much AI to write this. It's like much AI to write this. It's like the the LLMs of today just don't know the the LLMs of today just don't know the the LLMs of today just don't know this material very well because it's so this material very well because it's so this material very well because it's so new. new. new. But But But my goal for this book, number one, was my goal for this book, number one, was my goal for this book, number one, was that anyone would be able to pick it up that anyone would be able to pick it up that anyone would be able to pick it up and read the first 20 pages and and read the first 20 pages and and read the first 20 pages and understand my vision for where the understand my vision for where the understand my vision for where the future of AI is going. future of AI is going. future of AI is going. And then number two, if they, you know, And then number two, if they, you know, And then number two, if they, you know, make it through those 20 pages and make it through those 20 pages and make it through those 20 pages and they're really excited about the topic, they're really excited about the topic, they're really excited about the topic, that, you know, any smart person with a that, you know, any smart person with a that, you know, any smart person with a decent technical background is is going decent technical background is is going decent technical background is is going to be able to make it through the rest. to be able to make it through the rest. to be able to make it through the rest. And that meant making some trade-offs. And that meant making some trade-offs. And that meant making some trade-offs. You know, I I've definitely heard one of You know, I I've definitely heard one of You know, I I've definitely heard one of the I think most valid criticisms of the I think most valid criticisms of the I think most valid criticisms of this book is that it's a mile wide and this book is that it's a mile wide and this book is that it's a mile wide and an inch deep. It's very much a survey of an inch deep. It's very much a survey of an inch deep. It's very much a survey of all of these technologies. It doesn't all of these technologies. It doesn't all of these technologies. It doesn't necessarily teach you how to go hands-on necessarily teach you how to go hands-on necessarily teach you how to go hands-on and step-by-step do something. It's It's and step-by-step do something. It's It's and step-by-step do something. It's It's not one of your thousand-page books with not one of your thousand-page books with not one of your thousand-page books with a ton of code samples and a ton of a ton of code samples and a ton of a ton of code samples and a ton of hands-on guidance. And the reason I made hands-on guidance. And the reason I made hands-on guidance. And the reason I made that decision is partially just because that decision is partially just because that decision is partially just because I wanted a book that I could write very I wanted a book that I could write very I wanted a book that I could write very quickly. And partially because the goal quickly. And partially because the goal quickly. And partially because the goal of this book is more to teach someone of this book is more to teach someone of this book is more to teach someone what questions to ask rather than give what questions to ask rather than give what questions to ask rather than give them all of the answers. So, yeah, you them all of the answers. So, yeah, you them all of the answers. So, yeah, you know, it it is a dense book. It's a hard know, it it is a dense book. It's a hard know, it it is a dense book. It's a hard book, but like we have all of the sales book, but like we have all of the sales book, but like we have all of the sales people when they join Base 10, they they people when they join Base 10, they they people when they join Base 10, they they read this and and they get through it.

read this and and they get through it. read this and and they get through it. So, there's definitely, I think, a level So, there's definitely, I think, a level So, there's definitely, I think, a level of accessibility to the content that I of accessibility to the content that I of accessibility to the content that I prioritized. Interesting. Yeah, I mean, prioritized. Interesting. Yeah, I mean, prioritized. Interesting. Yeah, I mean, I did I did I did survey seems light to say that it is a survey seems light to say that it is a survey seems light to say that it is a survey. survey. survey. I mean, it is it is an overview surely I mean, it is it is an overview surely I mean, it is it is an overview surely of everything that is going on right now of everything that is going on right now of everything that is going on right now at this moment in time. And I'm sure at this moment in time. And I'm sure at this moment in time. And I'm sure that in two years, you should probably that in two years, you should probably that in two years, you should probably start working on or maybe a year, you start working on or maybe a year, you start working on or maybe a year, you should start working on the second should start working on the second should start working on the second edition. Absolutely. Right, because edition. Absolutely. Right, because edition. Absolutely. Right, because there that's the other irony of like there that's the other irony of like there that's the other irony of like having a physical book with actual pages having a physical book with actual pages having a physical book with actual pages and actual dead trees means that you and actual dead trees means that you and actual dead trees means that you know, how much of this is out of date. know, how much of this is out of date. know, how much of this is out of date. Not a lot. It's actually pretty pretty Not a lot. It's actually pretty pretty Not a lot. It's actually pretty pretty pretty fresh. pretty fresh. pretty fresh. But it did ask me it did give me a lot But it did ask me it did give me a lot But it did ask me it did give me a lot of questions to ask. And it's funny that of questions to ask. And it's funny that of questions to ask. And it's funny that you mentioned that because I'm looking you mentioned that because I'm looking you mentioned that because I'm looking here with my first question. here with my first question. here with my first question. I made it I made it I made it Let me see here. 41 pages in before I Let me see here. 41 pages in before I Let me see here. 41 pages in before I started dog-earing and looking stuff up. started dog-earing and looking stuff up. started dog-earing and looking stuff up. So, you said people if if smart people So, you said people if if smart people So, you said people if if smart people can make it 20 pages in, that's a good can make it 20 pages in, that's a good can make it 20 pages in, that's a good sign. I made it 41. Well, there you go.

sign. I made it 41. Well, there you go. sign. I made it 41. Well, there you go. >> So, I feel somewhat validated about >> So, I feel somewhat validated about >> So, I feel somewhat validated about this. this. this. Did Did you feel uncomfortable that you Did Did you feel uncomfortable that you Did Did you feel uncomfortable that you don't have a PhD in in machine learning don't have a PhD in in machine learning don't have a PhD in in machine learning and deep learning? Did you Did you have and deep learning? Did you Did you have and deep learning? Did you Did you have Did you Did you say like, "Uh should I Did you Did you say like, "Uh should I Did you Did you say like, "Uh should I be the one that writes the book?" How be the one that writes the book?" How be the one that writes the book?" How did you deal with that? Cuz I always did you deal with that? Cuz I always did you deal with that? Cuz I always feel like an impostor when I'm writing feel like an impostor when I'm writing feel like an impostor when I'm writing technical books. technical books. technical books. I am blessed with a complete absence of I am blessed with a complete absence of I am blessed with a complete absence of impostor syndrome. Oh, wow. You're the impostor syndrome. Oh, wow. You're the impostor syndrome. Oh, wow. You're the first. That's amazing. Tell us your first. That's amazing. Tell us your first. That's amazing. Tell us your ways. Perhaps some people in my life ways. Perhaps some people in my life ways. Perhaps some people in my life would would tell me that that a little would would tell me that that a little would would tell me that that a little bit of it might be good for me. But bit of it might be good for me. But bit of it might be good for me. But I think that there's a very academic I think that there's a very academic I think that there's a very academic take on AI that starts, you know, in the take on AI that starts, you know, in the take on AI that starts, you know, in the early days and defines what a perceptron early days and defines what a perceptron early days and defines what a perceptron is and teaches you what a convolutional is and teaches you what a convolutional is and teaches you what a convolutional neural network is and all that kind of neural network is and all that kind of neural network is and all that kind of stuff. And stuff. And stuff. And I entered the AI industry in 2022, I entered the AI industry in 2022, I entered the AI industry in 2022, January of 2022, about 10 months before January of 2022, about 10 months before January of 2022, about 10 months before ChatGPT became a huge thing. Mhm. And I ChatGPT became a huge thing. Mhm. And I ChatGPT became a huge thing. Mhm. And I didn't know any of that stuff.

didn't know any of that stuff. didn't know any of that stuff. And that hasn't stopped me from doing my And that hasn't stopped me from doing my And that hasn't stopped me from doing my job today. I'm sure all of that stuff is job today. I'm sure all of that stuff is job today. I'm sure all of that stuff is very valuable and I've enjoyed learning, very valuable and I've enjoyed learning, very valuable and I've enjoyed learning, you know, bits and pieces of it along you know, bits and pieces of it along you know, bits and pieces of it along the way. But I believe, and this is part the way. But I believe, and this is part the way. But I believe, and this is part of the thesis of of this project, that of the thesis of of this project, that of the thesis of of this project, that you can make a valuable contribution to you can make a valuable contribution to you can make a valuable contribution to the AI industry without a incredibly the AI industry without a incredibly the AI industry without a incredibly rigorous academic background. And that, rigorous academic background. And that, rigorous academic background. And that, you know, the the technologies of today you know, the the technologies of today you know, the the technologies of today are so new and moving so fast that in are so new and moving so fast that in are so new and moving so fast that in some ways it's an advantage to not know some ways it's an advantage to not know some ways it's an advantage to not know what you're doing because at least this what you're doing because at least this what you're doing because at least this way you don't have any bad habits to way you don't have any bad habits to way you don't have any bad habits to break. break. break. Yeah, that's interesting. It's funny Yeah, that's interesting. It's funny Yeah, that's interesting. It's funny that you mentioned that because I've that you mentioned that because I've that you mentioned that because I've I've talked about what it feels like to I've talked about what it feels like to I've talked about what it feels like to be towards the other side of your be towards the other side of your be towards the other side of your career, the end of your career. So, one career, the end of your career. So, one career, the end of your career. So, one could argue that like you're in the could argue that like you're in the could argue that like you're in the opening 10 years of your career and I'm opening 10 years of your career and I'm opening 10 years of your career and I'm in the latter 10 of mine. So then, in the latter 10 of mine. So then, in the latter 10 of mine. So then, should we, you know, should we shouldn't should we, you know, should we shouldn't should we, you know, should we shouldn't gatekeep this stuff just because gatekeep this stuff just because gatekeep this stuff just because someone's old or someone's young or someone's old or someone's young or someone's old or someone's young or someone came in at the beginning of the someone came in at the beginning of the someone came in at the beginning of the hockey stick growth within inference or hockey stick growth within inference or hockey stick growth within inference or someone, you know, started it at MIT or someone, you know, started it at MIT or someone, you know, started it at MIT or worked on the project in the 70s with worked on the project in the 70s with worked on the project in the 70s with so-and-so.

so-and-so. so-and-so. So, you're right. Like young people with So, you're right. Like young people with So, you're right. Like young people with fresh ideas and fresh perspectives are fresh ideas and fresh perspectives are fresh ideas and fresh perspectives are what's going to what's going to what's going to actually do the work and make this thing actually do the work and make this thing actually do the work and make this thing make this thing useful. make this thing useful. make this thing useful. So, one of the questions I wanted to ask So, one of the questions I wanted to ask So, one of the questions I wanted to ask about usefulness is this talks about about usefulness is this talks about about usefulness is this talks about hardware, talks about software, it talks hardware, talks about software, it talks hardware, talks about software, it talks about like I said, an overview of all about like I said, an overview of all about like I said, an overview of all the different GPUs, how they work, the different GPUs, how they work, the different GPUs, how they work, performance profiling. But underneath it performance profiling. But underneath it performance profiling. But underneath it is a reminder that like none of this is is a reminder that like none of this is is a reminder that like none of this is anything unless it actually does anything unless it actually does anything unless it actually does interesting and helpful work. interesting and helpful work. interesting and helpful work. Do you Do you think about the Do you Do you think about the Do you Do you think about the what can be done with AI? Because we what can be done with AI? Because we what can be done with AI? Because we opened this podcast by saying it can't opened this podcast by saying it can't opened this podcast by saying it can't write a book. And I don't know if I want write a book. And I don't know if I want write a book. And I don't know if I want it to write a book. Like I'm not it to write a book. Like I'm not it to write a book. Like I'm not interested in AI slop. I'm interested in interested in AI slop. I'm interested in interested in AI slop. I'm interested in AI to our point doing toil. So, I'm AI to our point doing toil. So, I'm AI to our point doing toil. So, I'm curious about your personal opinion. curious about your personal opinion. curious about your personal opinion. Like here we are with a book on AI Like here we are with a book on AI Like here we are with a book on AI inference, but it inference, but it inference, but it the book couldn't today be written by an the book couldn't today be written by an the book couldn't today be written by an AI and be any good. AI and be any good. AI and be any good. I think very little about vertical AI I think very little about vertical AI I think very little about vertical AI applications, actually. I mostly spend applications, actually. I mostly spend applications, actually. I mostly spend my cuz I spend so much of my day taking my cuz I spend so much of my day taking my cuz I spend so much of my day taking other people's vertical applications and other people's vertical applications and other people's vertical applications and helping them figure out how to make it helping them figure out how to make it helping them figure out how to make it twice as fast and half as expensive.

twice as fast and half as expensive. twice as fast and half as expensive. So, I will say that, you know, as a So, I will say that, you know, as a So, I will say that, you know, as a software engineer off and on throughout software engineer off and on throughout software engineer off and on throughout my career, obviously the past couple my career, obviously the past couple my career, obviously the past couple months have been very exciting and months have been very exciting and months have been very exciting and I feel like I have regained the ability I feel like I have regained the ability I feel like I have regained the ability to ship production code because there's to ship production code because there's to ship production code because there's some there's some trade-offs in in some there's some trade-offs in in some there's some trade-offs in in coming into taking six months and writing a book. In taking six months and writing a book. In that you your other skills can can that you your other skills can can that you your other skills can can atrophy a little bit. So, atrophy a little bit. So, atrophy a little bit. So, I find I find I find that I've been able to that I've been able to that I've been able to augment the sort of missing capabilities augment the sort of missing capabilities augment the sort of missing capabilities and ship the the stuff that I'm really and ship the the stuff that I'm really and ship the the stuff that I'm really excited about, no matter no matter where excited about, no matter no matter where excited about, no matter no matter where it is. And another thing is, you know, it is. And another thing is, you know, it is. And another thing is, you know, video editing. I do a lot of video video editing. I do a lot of video video editing. I do a lot of video editing with Descript because I'm not a editing with Descript because I'm not a editing with Descript because I'm not a fast or skilled editor. If anyone had a fast or skilled editor. If anyone had a fast or skilled editor. If anyone had a chance to watch my college YouTube chance to watch my college YouTube chance to watch my college YouTube videos before I put took those down a videos before I put took those down a videos before I put took those down a few years ago, uh they they would few years ago, uh they they would few years ago, uh they they would certainly know that I'm no Casey certainly know that I'm no Casey certainly know that I'm no Casey Neistat. Mhm. That's kind of the vision Neistat. Mhm. That's kind of the vision Neistat. Mhm. That's kind of the vision that I have for this space. Doesn't that I have for this space. Doesn't that I have for this space. Doesn't particularly matter what my vision is.

particularly matter what my vision is. particularly matter what my vision is. It matters the vision of the people who It matters the vision of the people who It matters the vision of the people who are actually building the tools, not are actually building the tools, not are actually building the tools, not just making them faster. just making them faster. just making them faster. But my my vision and my hope is that I'm But my my vision and my hope is that I'm But my my vision and my hope is that I'm going to be able to keep doing the going to be able to keep doing the going to be able to keep doing the things that I really love and outsource things that I really love and outsource things that I really love and outsource the pieces that I am not so good at and the pieces that I am not so good at and the pieces that I am not so good at and create things that otherwise I just create things that otherwise I just create things that otherwise I just would have left on my to-do list. would have left on my to-do list. would have left on my to-do list. That's a healthy attitude. I appreciate That's a healthy attitude. I appreciate That's a healthy attitude. I appreciate that. that. that. Like Like Like you're staying in your lane and you're you're staying in your lane and you're you're staying in your lane and you're apprecia- you you might dip your toes in apprecia- you you might dip your toes in apprecia- you you might dip your toes in another lane, but you know what you're another lane, but you know what you're another lane, but you know what you're good at and you stay away from the stuff good at and you stay away from the stuff good at and you stay away from the stuff that you're not. So, Base 10 is about that you're not. So, Base 10 is about that you're not. So, Base 10 is about speed, right? Like it's about making it speed, right? Like it's about making it speed, right? Like it's about making it better. And a lot of people concerned, better. And a lot of people concerned, better. And a lot of people concerned, myself I'm concerned about, you know, myself I'm concerned about, you know, myself I'm concerned about, you know, wasting energy on on AI. And you spend a wasting energy on on AI. And you spend a wasting energy on on AI. And you spend a lot of time in the book talking about lot of time in the book talking about lot of time in the book talking about performance. There's a ton like you you performance. There's a ton like you you performance. There's a ton like you you start right off the bat about like start right off the bat about like start right off the bat about like they're already making it faster. You they're already making it faster. You they're already making it faster. You talk about the difference between time talk about the difference between time talk about the difference between time to first token versus throughput. The to first token versus throughput. The to first token versus throughput. The the the whole thing is about throughput the the whole thing is about throughput the the whole thing is about throughput on honestly. If there was like a uh on honestly. If there was like a uh on honestly. If there was like a uh a through line through the book, it's a through line through the book, it's a through line through the book, it's like it's getting faster and it's going like it's getting faster and it's going like it's getting faster and it's going to get faster. How much faster do you to get faster. How much faster do you to get faster. How much faster do you think it can actually get? And is that think it can actually get? And is that think it can actually get? And is that going to be good in the sense of we'll going to be good in the sense of we'll going to be good in the sense of we'll waste less energy and it won't cost this waste less energy and it won't cost this waste less energy and it won't cost this many teaspoons of water to ask a many teaspoons of water to ask a many teaspoons of water to ask a question of a of a model.

question of a of a model. question of a of a model. So, with performance, So, with performance, So, with performance, there's a couple ways to think about it. there's a couple ways to think about it. there's a couple ways to think about it. It's a multi-variable optimization It's a multi-variable optimization It's a multi-variable optimization problem. But let's think just about problem. But let's think just about problem. But let's think just about three variables right now. Speed, cost three variables right now. Speed, cost three variables right now. Speed, cost cost and throughput are the same thing, cost and throughput are the same thing, cost and throughput are the same thing, and quality, the the correctness of of and quality, the the correctness of of and quality, the the correctness of of the output. the output. the output. So, quality is fixed at the model level. So, quality is fixed at the model level. So, quality is fixed at the model level. So, let's take the model as fixed for a So, let's take the model as fixed for a So, let's take the model as fixed for a second. So, now we only have to think second. So, now we only have to think second. So, now we only have to think about two variables, which are speed and about two variables, which are speed and about two variables, which are speed and cost or throughput. cost or throughput. cost or throughput. And you can plot an efficient frontier And you can plot an efficient frontier And you can plot an efficient frontier where on one axis you have speed, one where on one axis you have speed, one where on one axis you have speed, one axis you have cost. You can be maximally axis you have cost. You can be maximally axis you have cost. You can be maximally fast at a very high cost. You can be fast at a very high cost. You can be fast at a very high cost. You can be maximally cheap, but you're not going to maximally cheap, but you're not going to maximally cheap, but you're not going to be very fast. And then there's an be very fast. And then there's an be very fast. And then there's an efficient frontier curve between those efficient frontier curve between those efficient frontier curve between those two. two. two. Some model performance work is just Some model performance work is just Some model performance work is just figuring out where that efficient figuring out where that efficient figuring out where that efficient frontier is and helping you adjust the frontier is and helping you adjust the frontier is and helping you adjust the knobs and dials within your influence knobs and dials within your influence knobs and dials within your influence engine and whatnot to get to the place engine and whatnot to get to the place engine and whatnot to get to the place that you want to go on that frontier.

that you want to go on that frontier. that you want to go on that frontier. And then some performance work more on And then some performance work more on And then some performance work more on the research side is pushing that the research side is pushing that the research side is pushing that frontier out. And when you do that, when frontier out. And when you do that, when frontier out. And when you do that, when you create some kind of optimization you create some kind of optimization you create some kind of optimization that unlocks better performance, you can that unlocks better performance, you can that unlocks better performance, you can choose to move along move along the new choose to move along move along the new choose to move along move along the new frontier to either more speed at the frontier to either more speed at the frontier to either more speed at the same cost or same cost or same cost or or less cost at the same speed. or less cost at the same speed. or less cost at the same speed. So, So, So, then you add back the quality access, then you add back the quality access, then you add back the quality access, suddenly your efficient frontier becomes suddenly your efficient frontier becomes suddenly your efficient frontier becomes a sphere instead of a curved line. And a sphere instead of a curved line. And a sphere instead of a curved line. And as you either fine-tune a small as you either fine-tune a small as you either fine-tune a small open-source model to replace a large open-source model to replace a large open-source model to replace a large model, get a new performance frontier, model, get a new performance frontier, model, get a new performance frontier, find a spot along that, you you find a spot along that, you you find a spot along that, you you ultimately, like as this this research ultimately, like as this this research ultimately, like as this this research continues, you're able to push out and continues, you're able to push out and continues, you're able to push out and out your options around speed, cost, and out your options around speed, cost, and out your options around speed, cost, and quality. quality. quality. Again, kind of staying in my lane, like Again, kind of staying in my lane, like Again, kind of staying in my lane, like what I'm focused on is making the sphere what I'm focused on is making the sphere what I'm focused on is making the sphere as as big as possible, making it so that as as big as possible, making it so that as as big as possible, making it so that you have as much you have as much >> [snorts] >> sort of performance as a sort of >> sort of performance as a sort of >> sort of performance as a sort of a performance as a a performance as a a performance as a vague and term to spend in these vague and term to spend in these vague and term to spend in these trade-offs.

trade-offs. trade-offs. Where does that actually go? You know, Where does that actually go? You know, Where does that actually go? You know, we definitely see with inference getting we definitely see with inference getting we definitely see with inference getting cheaper and inference getting faster cheaper and inference getting faster cheaper and inference getting faster that the same amount of traffic that you that the same amount of traffic that you that the same amount of traffic that you maybe had 6 months ago, you can now maybe had 6 months ago, you can now maybe had 6 months ago, you can now serve with a fraction of the hardware or serve with a fraction of the hardware or serve with a fraction of the hardware or you can now serve for a fraction of the you can now serve for a fraction of the you can now serve for a fraction of the cost. You can now serve at a much better cost. You can now serve at a much better cost. You can now serve at a much better speed. speed. speed. But But But that is overshadowed by increases in that is overshadowed by increases in that is overshadowed by increases in demand. Um it's funny because a lot of demand. Um it's funny because a lot of demand. Um it's funny because a lot of the time we spend as a company is like the time we spend as a company is like the time we spend as a company is like trying to figure out how to get our trying to figure out how to get our trying to figure out how to get our customers to pay us less customers to pay us less customers to pay us less because if you can make your if you can because if you can make your if you can because if you can make your if you can make your inference X times faster or X make your inference X times faster or X make your inference X times faster or X times cheaper, you need fewer GPUs, times cheaper, you need fewer GPUs, times cheaper, you need fewer GPUs, there's less consumption, but that there's less consumption, but that there's less consumption, but that always gets rewarded with with more always gets rewarded with with more always gets rewarded with with more workloads and and more demand from the workloads and and more demand from the workloads and and more demand from the market. market. market. Right. So, at the at the sort of Right. So, at the at the sort of Right. So, at the at the sort of there's there's perhaps three layers to there's there's perhaps three layers to there's there's perhaps three layers to the answer to the question. There's the the answer to the question. There's the the answer to the question. There's the micro layer of on a workload by workload micro layer of on a workload by workload micro layer of on a workload by workload basis. Yes, we're making it faster and basis. Yes, we're making it faster and basis. Yes, we're making it faster and cheaper and less consumptive. And then cheaper and less consumptive. And then cheaper and less consumptive. And then at a macro level above that, maybe that at a macro level above that, maybe that at a macro level above that, maybe that increases demand, but at the macro level increases demand, but at the macro level increases demand, but at the macro level above that, well, the demand was there above that, well, the demand was there above that, well, the demand was there anyway. So, eventually we're going to anyway. So, eventually we're going to anyway. So, eventually we're going to just be able to fulfill it and we better just be able to fulfill it and we better just be able to fulfill it and we better do so in the most efficient way do so in the most efficient way do so in the most efficient way possible. Yeah. You know, there's the possible. Yeah. You know, there's the possible. Yeah. You know, there's the old computer engineering joke, which is old computer engineering joke, which is old computer engineering joke, which is you can have it good, fast, or cheap, you can have it good, fast, or cheap, you can have it good, fast, or cheap, pick two.

pick two. pick two. And there's always this idea that you And there's always this idea that you And there's always this idea that you can never have all three, but I think in can never have all three, but I think in can never have all three, but I think in the world of inference, in the world of the world of inference, in the world of the world of inference, in the world of AI, people expect and the market expects AI, people expect and the market expects AI, people expect and the market expects all three. all three. all three. And what I'm hearing you say is that you And what I'm hearing you say is that you And what I'm hearing you say is that you can optimize for two and then you'll can optimize for two and then you'll can optimize for two and then you'll bring that third one back up. So, like bring that third one back up. So, like bring that third one back up. So, like there will be good and fast and then there will be good and fast and then there will be good and fast and then maybe it won't be cheap, but then it maybe it won't be cheap, but then it maybe it won't be cheap, but then it will become cheap. And we are will become cheap. And we are will become cheap. And we are consistently seeing people using like consistently seeing people using like consistently seeing people using like hard math and hard computer science to hard math and hard computer science to hard math and hard computer science to make it good, fast, and cheap all at the make it good, fast, and cheap all at the make it good, fast, and cheap all at the same time. And then the the fourth one, same time. And then the the fourth one, same time. And then the the fourth one, and this is one that a lot of people and this is one that a lot of people and this is one that a lot of people don't really talk about, but it's don't really talk about, but it's don't really talk about, but it's actually very critical, is reliable. actually very critical, is reliable. actually very critical, is reliable. Because you have Because you have Because you have obviously all of the questions around obviously all of the questions around obviously all of the questions around the model reliability and you know, the the model reliability and you know, the the model reliability and you know, the prompting and the JSON mode output, prompting and the JSON mode output, prompting and the JSON mode output, structured output, function calling, structured output, function calling, structured output, function calling, whatever you're doing at the model level whatever you're doing at the model level whatever you're doing at the model level to make it reliable, setting that aside, to make it reliable, setting that aside, to make it reliable, setting that aside, there's the infrastructure reliability there's the infrastructure reliability there's the infrastructure reliability piece. How many nines of uptime are you piece. How many nines of uptime are you piece. How many nines of uptime are you getting from GPUs, which notoriously are getting from GPUs, which notoriously are getting from GPUs, which notoriously are not a very high nine piece of hardware.

not a very high nine piece of hardware. not a very high nine piece of hardware. You've got that, you've got the cloud You've got that, you've got the cloud You've got that, you've got the cloud provider reliability piece, you have the provider reliability piece, you have the provider reliability piece, you have the reliability of the inference reliability of the inference reliability of the inference optimizations that you're making. When optimizations that you're making. When optimizations that you're making. When you quantize, are you keeping quality you quantize, are you keeping quality you quantize, are you keeping quality intact? Is some novel speculation intact? Is some novel speculation intact? Is some novel speculation algorithm that you're introducing going algorithm that you're introducing going algorithm that you're introducing going to run into some niche bug and cause out to run into some niche bug and cause out to run into some niche bug and cause out of memory errors every so often? of memory errors every so often? of memory errors every so often? There's all kinds of applied research in There's all kinds of applied research in There's all kinds of applied research in the AI space that the AI space that the AI space that in other industries might take years or in other industries might take years or in other industries might take years or decades to make it into production, and decades to make it into production, and decades to make it into production, and here we go from, you know, paper to here we go from, you know, paper to here we go from, you know, paper to running on the GPU serving massive running on the GPU serving massive running on the GPU serving massive production traffic in weeks in many production traffic in weeks in many production traffic in weeks in many cases. So, there's a lot of work around cases. So, there's a lot of work around cases. So, there's a lot of work around like taking sort of research grade ideas like taking sort of research grade ideas like taking sort of research grade ideas and research grade software and and research grade software and and research grade software and hardening it to the point that you can hardening it to the point that you can hardening it to the point that you can serve a customer who has a four or five serve a customer who has a four or five serve a customer who has a four or five nine of uptime SLA because they're doing nine of uptime SLA because they're doing nine of uptime SLA because they're doing something critical like, you know, an an something critical like, you know, an an something critical like, you know, an an AI tool for doctors or some kind of, you AI tool for doctors or some kind of, you AI tool for doctors or some kind of, you know, financial tool.

know, financial tool. know, financial tool. The I'm going to shift gears a little The I'm going to shift gears a little The I'm going to shift gears a little bit cuz I'm thinking about like you like bit cuz I'm thinking about like you like bit cuz I'm thinking about like you like you're saying getting into production you're saying getting into production you're saying getting into production and I'm seeing people call themselves and I'm seeing people call themselves and I'm seeing people call themselves prompt engineers a lot and I prompt engineers a lot and I prompt engineers a lot and I kind of bugs me. I don't really like kind of bugs me. I don't really like kind of bugs me. I don't really like that term cuz like there's a there's a that term cuz like there's a there's a that term cuz like there's a there's a feeling that you can really control the feeling that you can really control the feeling that you can really control the black box by just poking at it with pros black box by just poking at it with pros black box by just poking at it with pros and I'm wondering if you can maybe and I'm wondering if you can maybe and I'm wondering if you can maybe juxtapose or explain the difference juxtapose or explain the difference juxtapose or explain the difference between prompt engineering and inference between prompt engineering and inference between prompt engineering and inference engineering. engineering. engineering. Absolutely. Uh we are building the Absolutely. Uh we are building the Absolutely. Uh we are building the playground that the prompt engineers playground that the prompt engineers playground that the prompt engineers hang out in. hang out in. hang out in. There is some overlap. So, for example, There is some overlap. So, for example, There is some overlap. So, for example, at the inference engine level, you can at the inference engine level, you can at the inference engine level, you can guarantee certain structured output guarantee certain structured output guarantee certain structured output pieces. Um if you look at back in 2023, pieces. Um if you look at back in 2023, pieces. Um if you look at back in 2023, 2024, there was a big movement around 2024, there was a big movement around 2024, there was a big movement around JSON output for models and and how do JSON output for models and and how do JSON output for models and and how do you get it to return an object. And you get it to return an object. And you get it to return an object. And there was for a long time a very prompt there was for a long time a very prompt there was for a long time a very prompt first view of this problem, which is first view of this problem, which is first view of this problem, which is like, you must return JSON and only JSON like, you must return JSON and only JSON like, you must return JSON and only JSON or my grandma's going to die.

or my grandma's going to die. or my grandma's going to die. And that's just not something that you And that's just not something that you And that's just not something that you want to bet your production workload on. want to bet your production workload on. want to bet your production workload on. So, now through the inference piece, we So, now through the inference piece, we So, now through the inference piece, we have the ability to guarantee a have the ability to guarantee a have the ability to guarantee a structured output. And the way that structured output. And the way that structured output. And the way that works is you define a schema, you send works is you define a schema, you send works is you define a schema, you send that in and along with your prompt, and that in and along with your prompt, and that in and along with your prompt, and what the inference engine does is it what the inference engine does is it what the inference engine does is it creates a state machine and it uses creates a state machine and it uses creates a state machine and it uses something called logit biasing. So, to something called logit biasing. So, to something called logit biasing. So, to dive into the the concepts of inference dive into the the concepts of inference dive into the the concepts of inference a little bit, and a large language model a little bit, and a large language model a little bit, and a large language model has something called a vocabulary, which has something called a vocabulary, which has something called a vocabulary, which is every single one of the 100,000 is every single one of the 100,000 is every single one of the 100,000 tokens it could possibly produce. tokens it could possibly produce. tokens it could possibly produce. And when it does a forward pass for And when it does a forward pass for And when it does a forward pass for inference to create that next token, inference to create that next token, inference to create that next token, what it actually does is it creates a what it actually does is it creates a what it actually does is it creates a vector of 100,000 probabilities of which vector of 100,000 probabilities of which vector of 100,000 probabilities of which token might be generated. token might be generated. token might be generated. With logit biasing, you do something With logit biasing, you do something With logit biasing, you do something called masking that vector. So, called masking that vector. So, called masking that vector. So, basically, you take every token that basically, you take every token that basically, you take every token that would not be valid syntax according to would not be valid syntax according to would not be valid syntax according to the state machine for your JSON schema the state machine for your JSON schema the state machine for your JSON schema and you just set the probability of that and you just set the probability of that and you just set the probability of that to zero or negative infinity depending to zero or negative infinity depending to zero or negative infinity depending on how you're how you're doing it. And on how you're how you're doing it. And on how you're how you're doing it. And then you still let the model do its then you still let the model do its then you still let the model do its generative thing, but you structurally generative thing, but you structurally generative thing, but you structurally guarantee that no invalid tokens are guarantee that no invalid tokens are guarantee that no invalid tokens are going to be generated.

going to be generated. going to be generated. And prompting of course in this case And prompting of course in this case And prompting of course in this case still matters because the contents of still matters because the contents of still matters because the contents of the JSON object that you're creating are the JSON object that you're creating are the JSON object that you're creating are not guaranteed by the schema. So, not guaranteed by the schema. So, not guaranteed by the schema. So, it being, you know, accurate and quality it being, you know, accurate and quality it being, you know, accurate and quality data is is still up to you in the data is is still up to you in the data is is still up to you in the prompting side, but the actual structure prompting side, but the actual structure prompting side, but the actual structure of the output is now guaranteed by the of the output is now guaranteed by the of the output is now guaranteed by the inference engine. So, that's an example inference engine. So, that's an example inference engine. So, that's an example of I mean, that's been the standard in of I mean, that's been the standard in of I mean, that's been the standard in production now for 18 to 24 months. production now for 18 to 24 months. production now for 18 to 24 months. But like that's an example of where the But like that's an example of where the But like that's an example of where the inference side can come in and and help inference side can come in and and help inference side can come in and and help the prompt side. But there's also places the prompt side. But there's also places the prompt side. But there's also places that the prompt side can come help the that the prompt side can come help the that the prompt side can come help the inference side. So, one of the biggest inference side. So, one of the biggest inference side. So, one of the biggest optimizations we do is KV cache reuse, optimizations we do is KV cache reuse, optimizations we do is KV cache reuse, where we're able to take the first few where we're able to take the first few where we're able to take the first few tokens of a prompt and reuse them tokens of a prompt and reuse them tokens of a prompt and reuse them between different requests as long as between different requests as long as between different requests as long as they match perfectly. they match perfectly. they match perfectly. So, for example, if you put your system So, for example, if you put your system So, for example, if you put your system prompt before your conversation or if prompt before your conversation or if prompt before your conversation or if you put all your shared context up front you put all your shared context up front you put all your shared context up front and then you change whatever's new about and then you change whatever's new about and then you change whatever's new about the specific request at the back of your the specific request at the back of your the specific request at the back of your prompt, your actual inference speed is prompt, your actual inference speed is prompt, your actual inference speed is going to be many times higher than if going to be many times higher than if going to be many times higher than if you put the novel stuff at the front of you put the novel stuff at the front of you put the novel stuff at the front of the prompt. So, there's there's a lot of the prompt. So, there's there's a lot of the prompt. So, there's there's a lot of interaction actually between the prompt interaction actually between the prompt interaction actually between the prompt and the inference layer. So, they, you and the inference layer. So, they, you and the inference layer. So, they, you know, they they have to work together to know, they they have to work together to know, they they have to work together to make a system completely optimized. I do make a system completely optimized. I do make a system completely optimized. I do think though that people think though that people think though that people over over pivot, especially when they're over over pivot, especially when they're over over pivot, especially when they're first starting building an actual first starting building an actual first starting building an actual production system, they over pivot on

production system, they over pivot on production system, they over pivot on like their prompts. I you see people on like their prompts. I you see people on like their prompts. I you see people on Twitter like, you know, this 20-line MD Twitter like, you know, this 20-line MD Twitter like, you know, this 20-line MD file is going to change the way your file is going to change the way your file is going to change the way your whole system works. It's like, putting whole system works. It's like, putting whole system works. It's like, putting don't do this in all caps don't do this in all caps don't do this in all caps has there's no guarantee that it's not has there's no guarantee that it's not has there's no guarantee that it's not going to do that. Like you just said, going to do that. Like you just said, going to do that. Like you just said, like, make sure that you return JSON, like, make sure that you return JSON, like, make sure that you return JSON, all caps. all caps. all caps. And it's like, yeah, but I said it in And it's like, yeah, but I said it in And it's like, yeah, but I said it in all caps and it still did it the other all caps and it still did it the other all caps and it still did it the other way. Like you really it is an ambiguity way. Like you really it is an ambiguity way. Like you really it is an ambiguity loop and you're going to get ambiguous loop and you're going to get ambiguous loop and you're going to get ambiguous results unless there's something results unless there's something results unless there's something probabilistic and and and and actual probabilistic and and and and actual probabilistic and and and and actual deterministic code deterministic code deterministic code that says this will only return JSON or, that says this will only return JSON or, that says this will only return JSON or, you know, like you you need a firewall, you know, like you you need a firewall, you know, like you you need a firewall, I guess is what I'm saying. Yeah. And I guess is what I'm saying. Yeah. And I guess is what I'm saying. Yeah. And the way you're describing that mask is the way you're describing that mask is the way you're describing that mask is it is a kind of a firewall. Like the it is a kind of a firewall. Like the it is a kind of a firewall. Like the machine's going to do what it's going to machine's going to do what it's going to machine's going to do what it's going to do, but we're not going to let bad do, but we're not going to let bad do, but we're not going to let bad results come out no matter what the results come out no matter what the results come out no matter what the prompt says. Exactly. It's it's the mix prompt says. Exactly. It's it's the mix prompt says. Exactly. It's it's the mix of using AI for what it's good at and of using AI for what it's good at and of using AI for what it's good at and using straight-up Python for what it's using straight-up Python for what it's using straight-up Python for what it's good at. So, like, for example, when I good at. So, like, for example, when I good at. So, like, for example, when I was proofreading the book, you would was proofreading the book, you would was proofreading the book, you would think models would be great at think models would be great at think models would be great at proofreading. They're not. And one of proofreading. They're not. And one of proofreading. They're not. And one of the things that makes them not good at the things that makes them not good at the things that makes them not good at it though is if you dump 47,000 words it though is if you dump 47,000 words it though is if you dump 47,000 words into a single chat window, uh it's it's into a single chat window, uh it's it's into a single chat window, uh it's it's not going to be able to go in and catch not going to be able to go in and catch not going to be able to go in and catch every error. But if you write a script every error. But if you write a script every error. But if you write a script that sends the book to the model a page that sends the book to the model a page that sends the book to the model a page at a time, uh you're going to get, you at a time, uh you're going to get, you at a time, uh you're going to get, you know, much better results. So, there's know, much better results. So, there's know, much better results. So, there's there's there's that aspect of it as there's there's that aspect of it as there's there's that aspect of it as well. It's just like knowing how these well. It's just like knowing how these well. It's just like knowing how these systems work under the hood makes you systems work under the hood makes you systems work under the hood makes you better at using them

better at using them better at using them and helps you build systems that get and helps you build systems that get and helps you build systems that get better results. See, this then you would better results. See, this then you would better results. See, this then you would love the analogy that I know my love the analogy that I know my love the analogy that I know my listeners are sick of me using, which is listeners are sick of me using, which is listeners are sick of me using, which is the learn how to drive stick shift. If the learn how to drive stick shift. If the learn how to drive stick shift. If you learn how to drive stick shift, you you learn how to drive stick shift, you you learn how to drive stick shift, you have a different relationship with the have a different relationship with the have a different relationship with the vehicle and I think a book like this is vehicle and I think a book like this is vehicle and I think a book like this is very much here's how the internal very much here's how the internal very much here's how the internal combustion engine works and now that you combustion engine works and now that you combustion engine works and now that you know how to drive stick, know how to drive stick, know how to drive stick, go back off and think about how to move go back off and think about how to move go back off and think about how to move people around and think about the people around and think about the people around and think about the concepts around transportation but but concepts around transportation but but concepts around transportation but but always have the engine and what's always have the engine and what's always have the engine and what's happening in inference engineering happening in inference engineering happening in inference engineering underneath so that you you you you know underneath so that you you you you know underneath so that you you you you know what you can actually affect in the what you can actually affect in the what you can actually affect in the larger system. Scott, that would be a larger system. Scott, that would be a larger system. Scott, that would be a fantastic metaphor, but I don't know how fantastic metaphor, but I don't know how fantastic metaphor, but I don't know how to drive. So to drive. So to drive. So >> Oh my god, you're killing me. >> Oh my god, you're killing me. >> Oh my god, you're killing me. >> guy. >> guy. >> guy. >> We're ending We're end See, this is the >> We're ending We're end See, this is the >> We're ending We're end See, this is the thing. thing. thing. >> [laughter] >> [laughter] >> [laughter] >> I love this. We're ending the show where >> I love this. We're ending the show where >> I love this. We're ending the show where you're a Waymo person. But see, the you're a Waymo person. But see, the you're a Waymo person. But see, the system requires you to know how to drive system requires you to know how to drive system requires you to know how to drive the way So the Waymo breaks down, you're the way So the Waymo breaks down, you're the way So the Waymo breaks down, you're going to get eaten by the zombies first going to get eaten by the zombies first going to get eaten by the zombies first is what I'm hearing. And I'm going to is what I'm hearing. And I'm going to is what I'm hearing. And I'm going to jump in, I'm going to hot-wire the car, jump in, I'm going to hot-wire the car, jump in, I'm going to hot-wire the car, throw it into first gear, and zip away.

throw it into first gear, and zip away. throw it into first gear, and zip away. That's true. That's true. But at least I That's true. That's true. But at least I That's true. That's true. But at least I I I understand when the Waymo makes a I I understand when the Waymo makes a I I understand when the Waymo makes a mistake why it's making it. So it it mistake why it's making it. So it it mistake why it's making it. So it it just, you know, gives me a little bit just, you know, gives me a little bit just, you know, gives me a little bit more empathy for the machine. more empathy for the machine. more empathy for the machine. This is why when the zombies come, we This is why when the zombies come, we This is why when the zombies come, we team up. team up. team up. We team up. It's a Philip It's a It's a We team up. It's a Philip It's a It's a We team up. It's a Philip It's a It's a It's a Philip and Hanselman partnership It's a Philip and Hanselman partnership It's a Philip and Hanselman partnership where I will drive stick, you will where I will drive stick, you will where I will drive stick, you will handle the Waymo. Between us, the handle the Waymo. Between us, the handle the Waymo. Between us, the zombies will not get us. zombies will not get us. zombies will not get us. >> [laughter] >> [laughter] >> [laughter] >> You're You're welcome on my zombie >> You're You're welcome on my zombie >> You're You're welcome on my zombie apocalypse squad anytime. I I I do also apocalypse squad anytime. I I I do also apocalypse squad anytime. I I I do also bring about 20 years of martial arts bring about 20 years of martial arts bring about 20 years of martial arts experience. That's my, you know, primary experience. That's my, you know, primary experience. That's my, you know, primary zombie apocalypse contribution. zombie apocalypse contribution. zombie apocalypse contribution. >> Very good. Well, I have a black belt in >> Very good. Well, I have a black belt in >> Very good. Well, I have a black belt in taekwondo, but I'm more of a slap taekwondo, but I'm more of a slap taekwondo, but I'm more of a slap fighter. So I will be standing behind fighter. So I will be standing behind fighter. So I will be standing behind you uh when the zombies come. Well, this you uh when the zombies come. Well, this you uh when the zombies come. Well, this is This has been fun and uh mean is This has been fun and uh mean is This has been fun and uh mean congratulations. This is a It's hard to congratulations. This is a It's hard to congratulations. This is a It's hard to write a book. It's hard to ship a book, write a book. It's hard to ship a book, write a book. It's hard to ship a book, and it's hard to get it into people's and it's hard to get it into people's and it's hard to get it into people's hands. And uh you all and the folks that hands. And uh you all and the folks that hands. And uh you all and the folks that you work with at Base Ten, like you you you work with at Base Ten, like you you you work with at Base Ten, like you you pulled it off. Thank you. It is very pulled it off. Thank you. It is very pulled it off. Thank you. It is very hard to get it in people's hands. I have hard to get it in people's hands. I have hard to get it in people's hands. I have made best friends with the members of my made best friends with the members of my made best friends with the members of my local FedEx local FedEx local FedEx uh office. Yeah, it is it is hard work.

uh office. Yeah, it is it is hard work. uh office. Yeah, it is it is hard work. >> Office 4096 in San Francisco. Highly >> Office 4096 in San Francisco. Highly >> Office 4096 in San Francisco. Highly recommended. Shout out. Shout out the recommended. Shout out. Shout out the recommended. Shout out. Shout out the crew there. There's definitely a lot to crew there. There's definitely a lot to crew there. There's definitely a lot to learn and also just another thing I just learn and also just another thing I just learn and also just another thing I just wanted to say like I really thought that wanted to say like I really thought that wanted to say like I really thought that the QR codes at the end were just such a the QR codes at the end were just such a the QR codes at the end were just such a nice touch. I I I just hate the idea nice touch. I I I just hate the idea nice touch. I I I just hate the idea that I'm going to get a book and have to that I'm going to get a book and have to that I'm going to get a book and have to type the URLs in and there's a huge type the URLs in and there's a huge type the URLs in and there's a huge appendix. Like honestly, there's appendix. Like honestly, there's appendix. Like honestly, there's probably 50 pages of just good like probably 50 pages of just good like probably 50 pages of just good like there's a glossary. It's just a there's a glossary. It's just a there's a glossary. It's just a thoughtful thing. So yeah, maybe the thoughtful thing. So yeah, maybe the thoughtful thing. So yeah, maybe the haters say that it's it's it's not deep. haters say that it's it's it's not deep. haters say that it's it's it's not deep. I found it to be just the right amount I found it to be just the right amount I found it to be just the right amount of deepness of deepness of deepness uh as a great survey for inference uh as a great survey for inference uh as a great survey for inference engineering and I enjoyed it very much engineering and I enjoyed it very much engineering and I enjoyed it very much and I appreciate you sending me a copy. and I appreciate you sending me a copy. and I appreciate you sending me a copy. Well, fantastic, Scott. That that that Well, fantastic, Scott. That that that Well, fantastic, Scott. That that that means a lot coming from you. means a lot coming from you. means a lot coming from you. I'm glad. So you can go ahead and check I'm glad. So you can go ahead and check I'm glad. So you can go ahead and check them out at base 10.co. That's b a s e t them out at base 10.co. That's b a s e t them out at base 10.co. That's b a s e t e n.co. Right at the top there, you can e n.co. Right at the top there, you can e n.co. Right at the top there, you can get your copy, get your digital copy, get your copy, get your digital copy, get your copy, get your digital copy, and then there's a waitlist now for the and then there's a waitlist now for the and then there's a waitlist now for the paper copies because they're hard to get paper copies because they're hard to get paper copies because they're hard to get and they're super popular. and they're super popular. and they're super popular. They are and Amazon, despite the fact They are and Amazon, despite the fact They are and Amazon, despite the fact that I have, you know, thousands and that I have, you know, thousands and that I have, you know, thousands and thousands of people now signed up to get thousands of people now signed up to get thousands of people now signed up to get paper copies, uh doesn't seem to want to paper copies, uh doesn't seem to want to paper copies, uh doesn't seem to want to list it. So we're going to we're going list it. So we're going to we're going list it. So we're going to we're going to have to find a way around that. Well, to have to find a way around that. Well, to have to find a way around that. Well, hopefully you'll figure it out soon.

hopefully you'll figure it out soon. hopefully you'll figure it out soon. Thank you so much, Philip Calley, for Thank you so much, Philip Calley, for Thank you so much, Philip Calley, for chatting with me today. chatting with me today. chatting with me today. Thank you, Scott. Have a good one. This Thank you, Scott. Have a good one. This Thank you, Scott. Have a good one. This has been another episode of has been another episode of has been another episode of Hanselminutes and we'll see you again Hanselminutes and we'll see you again Hanselminutes and we'll see you again next week.