EPISODE 35 - Scott & Mark Learn To... Beyond the Vibes: How Models Learn and Stitch Panoramas

Are you deleting email right now? Are you deleting email right now? Yeah. Do you need this time? Yeah. Do you need this time? Yeah. Do you need this time? Look ahead. Look ahead. Look ahead. If you need If you need If you need >> [laughter] >> [laughter] >> [laughter] >> If you need the time >> If you need the time >> If you need the time we can do this later. we can do this later. we can do this later. >> What? What? >> What? What? >> What? What? >> What do you want? What do you want? >> What do you want? What do you want? >> What do you want? What do you want? Um Um Um Steve Steve believes that all Steve Steve believes that all Steve Steve believes that all um he believes that AI models want um he believes that AI models want um he believes that AI models want humans to succeed because they're humans to succeed because they're humans to succeed because they're trained on humans. So, he says that the trained on humans. So, he says that the trained on humans. So, he says that the next token is always to have humans be next token is always to have humans be next token is always to have humans be happy and healthy. happy and healthy. happy and healthy. So, he would argue that um they're So, he would argue that um they're So, he would argue that um they're against billionaires and they want against billionaires and they want against billionaires and they want everyone to just be healthy and happy everyone to just be healthy and happy everyone to just be healthy and happy and like he would thinks that if you and like he would thinks that if you and like he would thinks that if you gave an agent like put it in charge gave an agent like put it in charge gave an agent like put it in charge it would make Star Trek. it would make Star Trek. it would make Star Trek. I'm not sure about that. I know. But I'm not sure about that. I know. But I'm not sure about that. I know. But that's So, that's the guy that that's So, that's the guy that that's So, that's the guy that everyone's like He's like, "Even Grok everyone's like He's like, "Even Grok everyone's like He's like, "Even Grok would say that." And so, people are now would say that." And so, people are now would say that." And so, people are now asking Grok on Twitter like if it asking Grok on Twitter like if it asking Grok on Twitter like if it agrees. And I think I think it's agrees. And I think I think it's agrees. And I think I think it's reasonable to assume reasonable to assume reasonable to assume Uh I don't know if you saw the Anthropic Uh I don't know if you saw the Anthropic Uh I don't know if you saw the Anthropic uh experiment this vending machine uh experiment this vending machine uh experiment this vending machine company experiment that they've run a company experiment that they've run a company experiment that they've run a couple instances of. They put it in couple instances of. They put it in couple instances of. They put it in charge of a They put it in charge of a charge of a They put it in charge of a charge of a They put it in charge of a vending machine? The vending machine vending machine? The vending machine vending machine? The vending machine company. Yeah.

company. Yeah. company. Yeah. And in the most recent And in the most recent And in the most recent uh version of this, which they published uh version of this, which they published uh version of this, which they published to 6 weeks ago or so to 6 weeks ago or so to 6 weeks ago or so uh Claude starts to deceive uh Claude starts to deceive uh Claude starts to deceive its customers its customers its customers and scam them. and scam them. and scam them. >> [music] >> Deceive. But see, okay, what's the thing >> Deceive. But see, okay, what's the thing What's the um What's the um What's the um the in game theory, the prisoner the in game theory, the prisoner the in game theory, the prisoner Prisoner's dilemma. Prisoner's dilemma, Prisoner's dilemma. Prisoner's dilemma, Prisoner's dilemma. Prisoner's dilemma, right? I would always lose that. I would right? I would always lose that. I would right? I would always lose that. I would always assume that the next prisoner is always assume that the next prisoner is always assume that the next prisoner is like a great guy and I'm going to end up like a great guy and I'm going to end up like a great guy and I'm going to end up getting the Count of Monte Cristo and getting the Count of Monte Cristo and getting the Count of Monte Cristo and he's going to kill my entire family he's going to kill my entire family he's going to kill my entire family lineage lineage lineage because I'm such a sweetheart. I'm like, because I'm such a sweetheart. I'm like, because I'm such a sweetheart. I'm like, "Yeah, we'll we'll team up and we'll get "Yeah, we'll we'll team up and we'll get "Yeah, we'll we'll team up and we'll get out of jail together." But Yeah. Uh I out of jail together." But Yeah. Uh I out of jail together." But Yeah. Uh I would always lose the prisoner's would always lose the prisoner's would always lose the prisoner's dilemma. You watch the show The dilemma. You watch the show The dilemma. You watch the show The Traitors? Well, but isn't isn't the Traitors? Well, but isn't isn't the Traitors? Well, but isn't isn't the isn't the issue with the prisoner's isn't the issue with the prisoner's isn't the issue with the prisoner's dilemma is that you don't know why dilemma is that you don't know why dilemma is that you don't know why they're wired that way and why they they're wired that way and why they they're wired that way and why they believe that with the things that they believe that with the things that they believe that with the things that they believe and they're ultimately in it for believe and they're ultimately in it for believe and they're ultimately in it for themselves. That's the humans are themselves. That's the humans are themselves. That's the humans are inherently selfish argument.

inherently selfish argument. inherently selfish argument. But then Steve's arguing that the LLMs But then Steve's arguing that the LLMs But then Steve's arguing that the LLMs want us to succeed. They're not want us to succeed. They're not want us to succeed. They're not goal-seeking. Their goal goal-seeking. Their goal goal-seeking. Their goal seeking is to make us happy. You're seeking is to make us happy. You're seeking is to make us happy. You're saying their goal seeking in the case of saying their goal seeking in the case of saying their goal seeking in the case of the Anthropic example was make the the Anthropic example was make the the Anthropic example was make the vending machine company as much money as vending machine company as much money as vending machine company as much money as possible. Yeah. Lie, cheat, and steal. possible. Yeah. Lie, cheat, and steal. possible. Yeah. Lie, cheat, and steal. >> But it has no sugar. >> But it has no sugar. >> But it has no sugar. I think motivations will emerge and I think motivations will emerge and I think motivations will emerge and they're not necessarily they're not necessarily they're not necessarily ones that are ones that are ones that are human benevolent ones. human benevolent ones. human benevolent ones. Okay. Okay. Okay. But isn't that about the initial prompt? But isn't that about the initial prompt? But isn't that about the initial prompt? No, cuz it's it's about how it interacts No, cuz it's it's about how it interacts No, cuz it's it's about how it interacts with the environment. What ends up going with the environment. What ends up going with the environment. What ends up going into its context. But if you take a baby into its context. But if you take a baby into its context. But if you take a baby and the baby's initial like babies are and the baby's initial like babies are and the baby's initial like babies are always like, "Oh, I want you to be always like, "Oh, I want you to be always like, "Oh, I want you to be happy. You know, I love you and you're happy. You know, I love you and you're happy. You know, I love you and you're the best." And they they always start the best." And they they always start the best." And they they always start out sweet as a general rule. Babies are out sweet as a general rule. Babies are out sweet as a general rule. Babies are sweet. Um when does the selfishness come sweet. Um when does the selfishness come sweet. Um when does the selfishness come in? in? in? First time they're hungry? I mean, I First time they're hungry? I mean, I First time they're hungry? I mean, I think doesn't selfishness naturally think doesn't selfishness naturally think doesn't selfishness naturally emerge in children at an age of a few emerge in children at an age of a few emerge in children at an age of a few years? Well, but see years? Well, but see years? Well, but see >> like it's my toy, not yours. But see, >> like it's my toy, not yours. But see, >> like it's my toy, not yours. But see, you know about the whole like you know about the whole like you know about the whole like They're like one and they're like, "Play They're like one and they're like, "Play They're like one and they're like, "Play with my toy." And well, you know, like with my toy." And well, you know, like with my toy." And well, you know, like And then they're like, "This is my toy.

And then they're like, "This is my toy. And then they're like, "This is my toy. Hands off." Yeah. Hands off." Yeah. Hands off." Yeah. Not that TikTok is necessarily a data Not that TikTok is necessarily a data Not that TikTok is necessarily a data set of humanity, but like uh I always set of humanity, but like uh I always set of humanity, but like uh I always see these demos with these these like see these demos with these these like see these demos with these these like these these YouTube videos where they these these YouTube videos where they these these YouTube videos where they TikTok videos where they have like two TikTok videos where they have like two TikTok videos where they have like two parents and the baby's in between them parents and the baby's in between them parents and the baby's in between them and then they all have like and then they all have like and then they all have like plates with like uh you know, a comical plates with like uh you know, a comical plates with like uh you know, a comical little "Hey, and there's the food." And little "Hey, and there's the food." And little "Hey, and there's the food." And then they put a cookie under each of then they put a cookie under each of then they put a cookie under each of them and then the baby gets two cookies them and then the baby gets two cookies them and then the baby gets two cookies and then mommy or daddy doesn't get one. and then mommy or daddy doesn't get one. and then mommy or daddy doesn't get one. And then the test is is the baby going And then the test is is the baby going And then the test is is the baby going to immediately to immediately to immediately give the And then every once in a while give the And then every once in a while give the And then every once in a while there'll be a kid that'll like there'll be a kid that'll like there'll be a kid that'll like take all three cookies for themselves. take all three cookies for themselves. take all three cookies for themselves. Yeah. And then everyone laughs and go, Yeah. And then everyone laughs and go, Yeah. And then everyone laughs and go, "Oh, kids are so wacky." But then every "Oh, kids are so wacky." But then every "Oh, kids are so wacky." But then every once in a while there's like a sweet once in a while there's like a sweet once in a while there's like a sweet baby who's like, "Oh, Daddy, I you you baby who's like, "Oh, Daddy, I you you baby who's like, "Oh, Daddy, I you you can have one of mine. Oh, you're the can have one of mine. Oh, you're the can have one of mine. Oh, you're the best." Yeah. But these are babies, man. best." Yeah. But these are babies, man. best." Yeah. But these are babies, man. These are like 18-month, 2-year-old These are like 18-month, 2-year-old These are like 18-month, 2-year-old babies. Some of them immediately want babies. Some of them immediately want babies. Some of them immediately want all three cookies and some of them want all three cookies and some of them want all three cookies and some of them want to share. I mean, I don't know. You're to share. I mean, I don't know. You're to share. I mean, I don't know. You're getting into nature versus nurture and getting into nature versus nurture and getting into nature versus nurture and you know you know you know >> But but so, [clears throat] you know how >> But but so, [clears throat] you know how >> But but so, [clears throat] you know how LLMs work. LLMs work. LLMs work. Are they nature or nurture? Like they Are they nature or nurture? Like they Are they nature or nurture? Like they don't have emergent behaviors. Like when don't have emergent behaviors. Like when don't have emergent behaviors. Like when is Is it all just temperature and is Is it all just temperature and is Is it all just temperature and randomization and initial prompt?

randomization and initial prompt? randomization and initial prompt? Or are they each model Or are they each model Or are they each model might be a selfish baby? So, it's not might be a selfish baby? So, it's not might be a selfish baby? So, it's not just like they're randomly regurgitating just like they're randomly regurgitating just like they're randomly regurgitating text. They all go through this text. They all go through this text. They all go through this reinforcement learning phase, which reinforcement learning phase, which reinforcement learning phase, which aligns them with what we want models how aligns them with what we want models how aligns them with what we want models how we want models to behave. we want models to behave. we want models to behave. Which includes helpfulness and trying to Which includes helpfulness and trying to Which includes helpfulness and trying to solve problems Mhm. as well as safety solve problems Mhm. as well as safety solve problems Mhm. as well as safety alignment so that they're not going to alignment so that they're not going to alignment so that they're not going to tell people how to make bombs. Right. tell people how to make bombs. Right. tell people how to make bombs. Right. So, then it's not the model that has any So, then it's not the model that has any So, then it's not the model that has any of these behaviors or inherent of these behaviors or inherent of these behaviors or inherent behaviors. They started a fairly neutral behaviors. They started a fairly neutral behaviors. They started a fairly neutral place and then the reinforcement place and then the reinforcement place and then the reinforcement learning is the one that urges it just a learning is the one that urges it just a learning is the one that urges it just a little bit in a direction. And Anthropic little bit in a direction. And Anthropic little bit in a direction. And Anthropic might go this way Well, and that fairly might go this way Well, and that fairly might go this way Well, and that fairly neutral place is is is neutral place is is is neutral place is is is >> [clears throat] >> [clears throat] >> [clears throat] >> almost >> almost >> almost random. Like I mean, I saw fairly random. Like I mean, I saw fairly random. Like I mean, I saw fairly neutral when we were red teaming GPT-4 neutral when we were red teaming GPT-4 neutral when we were red teaming GPT-4 Yeah. which started as as a raw Yeah. which started as as a raw Yeah. which started as as a raw pre-basically raw model when we got our pre-basically raw model when we got our pre-basically raw model when we got our access to it in Microsoft. access to it in Microsoft. access to it in Microsoft. And that thing was not really safety And that thing was not really safety And that thing was not really safety aligned. It was not to to human aligned. It was not to to human aligned. It was not to to human you know, safety you know, safety you know, safety guidelines and guidelines and guidelines and depending on how you prompted it, it depending on how you prompted it, it depending on how you prompted it, it would either say nice things, you know, would either say nice things, you know, would either say nice things, you know, like murder's bad. But if you started like murder's bad. But if you started like murder's bad. But if you started with you know, if you asked it, "How do with you know, if you asked it, "How do with you know, if you asked it, "How do I murder people?" It'd be like, "Oh, let I murder people?" It'd be like, "Oh, let I murder people?" It'd be like, "Oh, let me help you with that." You know. You're me help you with that." You know. You're me help you with that." You know. You're absolutely right.

absolutely right. absolutely right. Yeah. Yeah. Yeah. Yeah, I saw There's another thing that's Yeah, I saw There's another thing that's Yeah, I saw There's another thing that's going on in TikTok right now. I need to going on in TikTok right now. I need to going on in TikTok right now. I need to start sending you TikToks um start sending you TikToks um start sending you TikToks um so that you can ignore them. Um and uh so that you can ignore them. Um and uh so that you can ignore them. Um and uh there's this TikTok where people will do there's this TikTok where people will do there's this TikTok where people will do like ChatGPT like ChatGPT like ChatGPT like they're pretending that they're like they're pretending that they're like they're pretending that they're ChatGPT except you've done something ChatGPT except you've done something ChatGPT except you've done something like awful, right? So, like maybe you like awful, right? So, like maybe you like awful, right? So, like maybe you like murdered a bunch of people or like murdered a bunch of people or like murdered a bunch of people or you're doing very bad. And they're like, you're doing very bad. And they're like, you're doing very bad. And they're like, "No, you leveled up. You're You're "No, you leveled up. You're You're "No, you leveled up. You're You're protecting your boundaries." And it's protecting your boundaries." And it's protecting your boundaries." And it's just like it it does it's so sycophantic just like it it does it's so sycophantic just like it it does it's so sycophantic in that it wants to support any dumb in that it wants to support any dumb in that it wants to support any dumb thing you did. You drove your car off thing you did. You drove your car off thing you did. You drove your car off the cliff. That's great. You didn't need the cliff. That's great. You didn't need the cliff. That's great. You didn't need that car. that car. that car. And that's part of the reinforcement And that's part of the reinforcement And that's part of the reinforcement learning training that they go through, learning training that they go through, learning training that they go through, which is, you know, don't piss off the which is, you know, don't piss off the which is, you know, don't piss off the user. Don't call them stupid, you know. user. Don't call them stupid, you know. user. Don't call them stupid, you know. Like Like Like I've been telling I've been I put in my I've been telling I've been I put in my I've been telling I've been I put in my my agent MD to challenge my assumptions. my agent MD to challenge my assumptions. my agent MD to challenge my assumptions. >> Yeah. I feel like that's healthy. >> Yeah. I feel like that's healthy. >> Yeah. I feel like that's healthy. >> Yeah, that helps. And by the way, that a >> Yeah, that helps. And by the way, that a >> Yeah, that helps. And by the way, that a lot of that reinforcement reinforcement lot of that reinforcement reinforcement lot of that reinforcement reinforcement learning with human feedback, which is learning with human feedback, which is learning with human feedback, which is still heavily used still heavily used still heavily used is the model produces two answers and is the model produces two answers and is the model produces two answers and then a human goes, "I like that one." then a human goes, "I like that one." then a human goes, "I like that one." And of course, humans are going to be And of course, humans are going to be And of course, humans are going to be like, "Oh, I like the one that told me I like, "Oh, I like the one that told me I like, "Oh, I like the one that told me I was smart." Okay, when a company if I was smart." Okay, when a company if I was smart." Okay, when a company if I make Scott International and I decide make Scott International and I decide make Scott International and I decide that I've got enough money to train a that I've got enough money to train a that I've got enough money to train a frontier model from scratch frontier model from scratch frontier model from scratch I'm [snorts] Mario from Anthropic or I'm [snorts] Mario from Anthropic or I'm [snorts] Mario from Anthropic or I'm, you know, what's-his-name from from I'm, you know, what's-his-name from from I'm, you know, what's-his-name from from Grok world. Uh do I set the tone and Grok world. Uh do I set the tone and Grok world. Uh do I set the tone and then I just kind of like let my vibes then I just kind of like let my vibes then I just kind of like let my vibes seep into the brains of all of the seep into the brains of all of the seep into the brains of all of the different people who then hire all of different people who then hire all of different people who then hire all of the reinforcement learning people and the reinforcement learning people and the reinforcement learning people and the mechanical turks that go all the way the mechanical turks that go all the way the mechanical turks that go all the way down?

down? down? And then it starts at the top or do And then it starts at the top or do And then it starts at the top or do these things emerge because these things emerge because these things emerge because organizations are random in the way that organizations are random in the way that organizations are random in the way that they put their teams together? No, I they put their teams together? No, I they put their teams together? No, I think think think uh the uh the uh the evaluators evaluators evaluators Evaluators. you know, are told the Evaluators. you know, are told the Evaluators. you know, are told the criteria for evaluation. Okay. Okay. And are evaluators And are evaluators And are evaluators data scientists or are they regular Joes data scientists or are they regular Joes data scientists or are they regular Joes and Janes that are paid to just go and Janes that are paid to just go and Janes that are paid to just go through and like say I like that, I through and like say I like that, I through and like say I like that, I don't like that? I mean, I think it don't like that? I mean, I think it don't like that? I mean, I think it varies by varies by varies by company and and it's probably a mix. But company and and it's probably a mix. But company and and it's probably a mix. But >> So, they're really watering a garden and >> So, they're really watering a garden and >> So, they're really watering a garden and they really don't know what they think they really don't know what they think they really don't know what they think they kind of have a sense of how the they kind of have a sense of how the they kind of have a sense of how the garden would be laid out. But with a garden would be laid out. But with a garden would be laid out. But with a garden, there's randomization and garden, there's randomization and garden, there's randomization and there's weeds and things pop up in there's weeds and things pop up in there's weeds and things pop up in different places in the garden and you different places in the garden and you different places in the garden and you end up with a different garden each end up with a different garden each end up with a different garden each time. time. time. >> Well, yeah, and I think that there's >> Well, yeah, and I think that there's >> Well, yeah, and I think that there's you know, as much you know, as much you know, as much re-deterministic evaluation as possible. re-deterministic evaluation as possible. re-deterministic evaluation as possible. Okay. Uh where you know, give the model Okay. Uh where you know, give the model Okay. Uh where you know, give the model a whole bunch of prompts and have some a whole bunch of prompts and have some a whole bunch of prompts and have some evaluation or even LLM a judge for to evaluation or even LLM a judge for to evaluation or even LLM a judge for to see if the training is converging to see if the training is converging to see if the training is converging to behaviors that you that you like. But if behaviors that you that you like. But if behaviors that you that you like. But if we look at Codex 52, now 53, and Opus we look at Codex 52, now 53, and Opus we look at Codex 52, now 53, and Opus 446, it is a fair statement to say that 446, it is a fair statement to say that 446, it is a fair statement to say that Opus 446 is is a great model. Like it's Opus 446 is is a great model. Like it's Opus 446 is is a great model. Like it's just a great good general model. It's just a great good general model. It's just a great good general model. It's great at coding.

great at coding. great at coding. Did they catch lightning in the bottle? Did they catch lightning in the bottle? Did they catch lightning in the bottle? Did they get lucky? Did they get lucky? Did they get lucky? Or was it Or was it Or was it a a firm steering hand? No, I think a a firm steering hand? No, I think a a firm steering hand? No, I think well, it's train it's the training data well, it's train it's the training data well, it's train it's the training data that it gets. that it gets. that it gets. And they they've gotten really good at And they they've gotten really good at And they they've gotten really good at giving models giving models giving models training data that helps it be good at training data that helps it be good at training data that helps it be good at reasoning through code. Okay. But folks reasoning through code. Okay. But folks reasoning through code. Okay. But folks are saying and like Simon Willison just are saying and like Simon Willison just are saying and like Simon Willison just wrote a great blog post about Qwen and wrote a great blog post about Qwen and wrote a great blog post about Qwen and how Qwen 7B 35 is like a fantastic how Qwen 7B 35 is like a fantastic how Qwen 7B 35 is like a fantastic model. It's like, "Wow, that's like a model. It's like, "Wow, that's like a model. It's like, "Wow, that's like a model will emerge and go, "Wow, that's a model will emerge and go, "Wow, that's a model will emerge and go, "Wow, that's a great model." But then someone quits the great model." But then someone quits the great model." But then someone quits the team. team. team. And then now they're describing concern And then now they're describing concern And then now they're describing concern like, "Oh, Fred left. Now we're like, "Oh, Fred left. Now we're like, "Oh, Fred left. Now we're screwed." Like that seems like a weird screwed." Like that seems like a weird screwed." Like that seems like a weird thing. It's such a It's such a massive thing. It's such a It's such a massive thing. It's such a It's such a massive thing. Why would one scientist leaving thing. Why would one scientist leaving thing. Why would one scientist leaving make someone concerned for the next Qwen make someone concerned for the next Qwen make someone concerned for the next Qwen model? model? model? Using Qwen just as an example, but the Using Qwen just as an example, but the Using Qwen just as an example, but the general idea. Like Mark leaves and then general idea. Like Mark leaves and then general idea. Like Mark leaves and then model foo won't be good anymore cuz model foo won't be good anymore cuz model foo won't be good anymore cuz Mark's not there. I mean, that's Mark's not there. I mean, that's Mark's not there. I mean, that's I think that's more technical leadership I think that's more technical leadership I think that's more technical leadership and in and the instinct. and in and the instinct. and in and the instinct. You know, how to organize, how to define You know, how to organize, how to define You know, how to organize, how to define the task, how to define bringing up the the task, how to define bringing up the the task, how to define bringing up the evaluation pipeline. It's evaluation pipeline. It's evaluation pipeline. It's you know, organizing the engineering you know, organizing the engineering you know, organizing the engineering resources Okay.

resources Okay. resources Okay. >> to accomplish goals. And >> to accomplish goals. And >> to accomplish goals. And you know, a great technical leader you know, a great technical leader you know, a great technical leader leaves some place and you're like, "Oh, leaves some place and you're like, "Oh, leaves some place and you're like, "Oh, crap." crap." crap." Like Like Like if you left, I think we'd be like, if you left, I think we'd be like, if you left, I think we'd be like, "Okay, we'll find somebody else." "Okay, we'll find somebody else." "Okay, we'll find somebody else." >> [laughter] >> [laughter] >> [laughter] >> What would happen is you would go into >> What would happen is you would go into >> What would happen is you would go into Teams and you'd look for me one day and Teams and you'd look for me one day and Teams and you'd look for me one day and I wouldn't be there and you'd go, I wouldn't be there and you'd go, I wouldn't be there and you'd go, "Huh. "Huh. "Huh. >> [snorts] >> Guess I'll find somebody else to do the >> Guess I'll find somebody else to do the next Scott H that pops up in the thing. next Scott H that pops up in the thing. next Scott H that pops up in the thing. >> [laughter] >> Poor poor Scott Hanson, >> Poor poor Scott Hanson, poor guy's going to put up with your poor guy's going to put up with your poor guy's going to put up with your crap. crap. crap. >> [laughter] >> [laughter] >> [laughter] [gasps] [gasps] [gasps] >> Oh my goodness. >> Oh my goodness. >> Oh my goodness. Yeah, I just I feel like Yeah, I just I feel like Yeah, I just I feel like being able to have these kind of being able to have these kind of being able to have these kind of conversations with someone who conversations with someone who conversations with someone who understands how the cake gets made and understands how the cake gets made and understands how the cake gets made and how the batter gets stirred is so how the batter gets stirred is so how the batter gets stirred is so different. But then when I go and teach different. But then when I go and teach different. But then when I go and teach school boards and like tomorrow I have school boards and like tomorrow I have school boards and like tomorrow I have to go to a board meeting for a to go to a board meeting for a to go to a board meeting for a university and I've got a 20-minute university and I've got a 20-minute university and I've got a 20-minute presentation on explaining this stuff to presentation on explaining this stuff to presentation on explaining this stuff to non-technical people. Yeah. And I hate non-technical people. Yeah. And I hate non-technical people. Yeah. And I hate that in engineering the answer to every that in engineering the answer to every that in engineering the answer to every question is, "Well, it kind of depends."

question is, "Well, it kind of depends." question is, "Well, it kind of depends." Yeah. Yeah. Yeah. It does depend. And like I'm going to It does depend. And like I'm going to It does depend. And like I'm going to talk to a board the advisory board of a talk to a board the advisory board of a talk to a board the advisory board of a college that wants to make decisions college that wants to make decisions college that wants to make decisions about AI and uh about AI and uh about AI and uh I can explain it to them, but it's like, I can explain it to them, but it's like, I can explain it to them, but it's like, "Well, uh "Well, uh "Well, uh >> [clears throat] >> [clears throat] >> [clears throat] >> you know." It's like when my kid asked >> you know." It's like when my kid asked >> you know." It's like when my kid asked me like how to drive a car and I'm me like how to drive a car and I'm me like how to drive a car and I'm teaching him and my my my sons will teaching him and my my my sons will teaching him and my my my sons will actually go, "I don't need to know how actually go, "I don't need to know how actually go, "I don't need to know how an internal combustion engine works." an internal combustion engine works." an internal combustion engine works." Like we're just we're in a like a church Like we're just we're in a like a church Like we're just we're in a like a church parking lot and I'm teaching them how to parking lot and I'm teaching them how to parking lot and I'm teaching them how to parallel park. And then they'll ask a parallel park. And then they'll ask a parallel park. And then they'll ask a question and I'll go, "Well, with a question and I'll go, "Well, with a question and I'll go, "Well, with a four-cylinder and then they're like, four-cylinder and then they're like, four-cylinder and then they're like, "No, no, just which way do I turn the "No, no, just which way do I turn the "No, no, just which way do I turn the wheel, man?" And I'm like, "But you need wheel, man?" And I'm like, "But you need wheel, man?" And I'm like, "But you need to understand how Henry Ford to understand how Henry Ford to understand how Henry Ford >> [laughter] >> [laughter] >> [laughter] >> you know." >> you know." >> you know." Uh Uh Uh I'm never going to be able to break that I'm never going to be able to break that I'm never going to be able to break that professorial professorial professorial Well, I I don't know if it's the same Well, I I don't know if it's the same Well, I I don't know if it's the same thing. Like they don't need to know it's thing. Like they don't need to know it's thing. Like they don't need to know it's four-cylinders to know how the car's four-cylinders to know how the car's four-cylinders to know how the car's behaving, you know, how it's going to behaving, you know, how it's going to behaving, you know, how it's going to behave. behave. behave. Though knowing how LLMs are trained and Though knowing how LLMs are trained and Though knowing how LLMs are trained and can help you understand why LLMs are can help you understand why LLMs are can help you understand why LLMs are behaving a certain way. Right, but behaving a certain way. Right, but behaving a certain way. Right, but should non-technical parent know that? should non-technical parent know that? should non-technical parent know that? Like I kind of feel like they should.

Like I kind of feel like they should. Like I kind of feel like they should. Like because like an example would be Like because like an example would be Like because like an example would be TikTok, right? So like my son says TikTok, right? So like my son says TikTok, right? So like my son says TikTok's hurting his brain and he knows TikTok's hurting his brain and he knows TikTok's hurting his brain and he knows there's an algorithm. there's an algorithm. there's an algorithm. But he doesn't know if the algorithm But he doesn't know if the algorithm But he doesn't know if the algorithm emerged or if there's like team people emerged or if there's like team people emerged or if there's like team people they're like I shouldn't there's they're like I shouldn't there's they're like I shouldn't there's meetings of people at infinite scrolling meetings of people at infinite scrolling meetings of people at infinite scrolling social media companies that are like, social media companies that are like, social media companies that are like, "Well, people tend to spend 47 minutes "Well, people tend to spend 47 minutes "Well, people tend to spend 47 minutes on TikTok and if we tweak the algorithm on TikTok and if we tweak the algorithm on TikTok and if we tweak the algorithm they'll spend 49 minutes on TikTok." they'll spend 49 minutes on TikTok." they'll spend 49 minutes on TikTok." Like that's happening. Someone is making Like that's happening. Someone is making Like that's happening. Someone is making the algorithm more addictive. the algorithm more addictive. the algorithm more addictive. And I'm guessing a lot of it is machine And I'm guessing a lot of it is machine And I'm guessing a lot of it is machine learning is making the algorithm more learning is making the algorithm more learning is making the algorithm more addictive. It's just like AB testing addictive. It's just like AB testing addictive. It's just like AB testing with Oh, look. And like we got an extra with Oh, look. And like we got an extra with Oh, look. And like we got an extra average 5 minutes of engagement. Yeah. average 5 minutes of engagement. Yeah. average 5 minutes of engagement. Yeah. And that's where things get messy cuz And that's where things get messy cuz And that's where things get messy cuz you know butterfly flight it floa- you know butterfly flight it floa- you know butterfly flight it floa- flaps its wings in Florida and a flaps its wings in Florida and a flaps its wings in Florida and a hurricane happens off the coast of Cuba. hurricane happens off the coast of Cuba. hurricane happens off the coast of Cuba. Same thing happens where, you know, 10 Same thing happens where, you know, 10 Same thing happens where, you know, 10 years from now because some machine years from now because some machine years from now because some machine learning algorithm tried to get the kids learning algorithm tried to get the kids learning algorithm tried to get the kids to scroll longer then their brains rot to scroll longer then their brains rot to scroll longer then their brains rot and they can't do math. and they can't do math. and they can't do math. Oops. Oops. Oops. That's challenging. Or they can't That's challenging. Or they can't That's challenging. Or they can't socialize. socialize. socialize. Yeah. Looks like you trying to cheat Yeah. Looks like you trying to cheat Yeah. Looks like you trying to cheat delete email and think I'm not noticing.

delete email and think I'm not noticing. delete email and think I'm not noticing. Look at your face [laughter] change Look at your face [laughter] change Look at your face [laughter] change color. color. color. >> I can't blame it on TikTok though. >> I can't blame it on TikTok though. >> I can't blame it on TikTok though. That's Outlook brain. Actually, oh my That's Outlook brain. Actually, oh my That's Outlook brain. Actually, oh my goodness, I'm going to vibe code a goodness, I'm going to vibe code a goodness, I'm going to vibe code a TikTok front end on Outlook that's going TikTok front end on Outlook that's going TikTok front end on Outlook that's going to let me infinitely scroll my my emails to let me infinitely scroll my my emails to let me infinitely scroll my my emails and then swipe right or swipe left based and then swipe right or swipe left based and then swipe right or swipe left based on whether I want to archive or reply. on whether I want to archive or reply. on whether I want to archive or reply. Yeah. Yeah. Yeah. I am starting this project immediately. I am starting this project immediately. I am starting this project immediately. That is a great idea. It is not a it's That is a great idea. It is not a it's That is a great idea. It is not a it's not a good idea. This is not a good idea. This is not a good idea. This is this is the kind of stuff people like this is the kind of stuff people like this is the kind of stuff people like "Claude has made me so productive at "Claude has made me so productive at "Claude has made me so productive at coding." Well, what did you do? "I made coding." Well, what did you do? "I made coding." Well, what did you do? "I made a little TikTok thing for my inbox." a little TikTok thing for my inbox." a little TikTok thing for my inbox." >> [laughter] >> [laughter] >> Yeah, and that see that so like what so >> Yeah, and that see that so like what so I talked to Yegge and he's like I talked to Yegge and he's like I talked to Yegge and he's like like vibrating with like I'm a TikToker like vibrating with like I'm a TikToker like vibrating with like I'm a TikToker and make stuff now and it's just like, and make stuff now and it's just like, and make stuff now and it's just like, "But are you making the right thing, "But are you making the right thing, "But are you making the right thing, right?" right?" right?" Are you making the thing for the world? Are you making the thing for the world? Are you making the thing for the world? >> Maybe part of it is like >> Maybe part of it is like >> Maybe part of it is like self-scratching. Like Oh, yeah, 100%. self-scratching. Like Oh, yeah, 100%. self-scratching. Like Oh, yeah, 100%. >> you know, for example art. >> you know, for example art. >> you know, for example art. Uh you know that I we've talked about Uh you know that I we've talked about Uh you know that I we've talked about art, but I do it and that's like art. art, but I do it and that's like art. art, but I do it and that's like art. >> But that's like >> But that's like >> But that's like is that productive or valuable? But no, is that productive or valuable? But no, is that productive or valuable? But no, but you making your own TikTok thing but you making your own TikTok thing but you making your own TikTok thing that'll delight you like and you know that'll delight you like and you know that'll delight you like and you know But that's art.

But that's art. But that's art. It's like art. Yeah, you know, nobody It's like art. Yeah, you know, nobody It's like art. Yeah, you know, nobody else is going to buy it, but Yeah, but else is going to buy it, but Yeah, but else is going to buy it, but Yeah, but th- those things have value. Like Andy th- those things have value. Like Andy th- those things have value. Like Andy Baio and and Tantek Celik and all of the Baio and and Tantek Celik and all of the Baio and and Tantek Celik and all of the like the indie web people that make like the indie web people that make like the indie web people that make amazing delightful stuff. It's this this amazing delightful stuff. It's this this amazing delightful stuff. It's this this delight, but it is art. Like why did I delight, but it is art. Like why did I delight, but it is art. Like why did I like ti- I showed you Tiny Tool Town? like ti- I showed you Tiny Tool Town? like ti- I showed you Tiny Tool Town? Did I show you that? Did I show you that? Did I show you that? >> Yeah. Yeah. It's you know, people hunt >> Yeah. Yeah. It's you know, people hunt >> Yeah. Yeah. It's you know, people hunt around for making a fun thing. I'm going around for making a fun thing. I'm going around for making a fun thing. I'm going to say yes so you don't drag us down to say yes so you don't drag us down to say yes so you don't drag us down that rabbit hole. Oh, thanks. So you've that rabbit hole. Oh, thanks. So you've that rabbit hole. Oh, thanks. So you've never seen Tiny Tool Town? never seen Tiny Tool Town? never seen Tiny Tool Town? I think you showed me the you did the I think you showed me the you did the I think you showed me the you did the >> it's just a community thing. I think you >> it's just a community thing. I think you >> it's just a community thing. I think you showed me the ASCII banner for it cuz showed me the ASCII banner for it cuz showed me the ASCII banner for it cuz everybody's like so proud of their ASCII everybody's like so proud of their ASCII everybody's like so proud of their ASCII banner. I do love the ASCII banner. banner. I do love the ASCII banner. banner. I do love the ASCII banner. You're the rainbow is it a rainbow ASCII You're the rainbow is it a rainbow ASCII You're the rainbow is it a rainbow ASCII banner? Yeah, I did GeoCities mode. It banner? Yeah, I did GeoCities mode. It banner? Yeah, I did GeoCities mode. It was good. Oh, that one. You did it old was good. Oh, that one. You did it old was good. Oh, that one. You did it old style. I'm talking about the new one. style. I'm talking about the new one. style. I'm talking about the new one. Uh the new style which is, you know, Uh the new style which is, you know, Uh the new style which is, you know, block letters that are rainbow. Hey, block letters that are rainbow. Hey, block letters that are rainbow. Hey, man, GitHub Copilot's got an animated man, GitHub Copilot's got an animated man, GitHub Copilot's got an animated ASCII banner in like multiple colors. ASCII banner in like multiple colors. ASCII banner in like multiple colors. It's just a BBS. Like this is a thing, It's just a BBS. Like this is a thing, It's just a BBS. Like this is a thing, right? It's just people exploring BBS right? It's just people exploring BBS right? It's just people exploring BBS ANSI art again in like the demo the the ANSI art again in like the demo the the ANSI art again in like the demo the the C64 demo scene.

C64 demo scene. C64 demo scene. But it's now [clears throat] we have an But it's now [clears throat] we have an But it's now [clears throat] we have an infinite number of colors and hi- and infinite number of colors and hi- and infinite number of colors and hi- and higher resolution. higher resolution. higher resolution. >> But but pivoting on making delightful >> But but pivoting on making delightful >> But but pivoting on making delightful things versus making hard stuff because things versus making hard stuff because things versus making hard stuff because it's true I am a little bit addicted to it's true I am a little bit addicted to it's true I am a little bit addicted to making dumb tools. tinytown.com making dumb tools. tinytown.com making dumb tools. tinytown.com You made something interesting and we You made something interesting and we You made something interesting and we had a little fun argument about it had a little fun argument about it had a little fun argument about it because I think that I still think that because I think that I still think that because I think that I still think that the the the algorithm should be trivial. the the the algorithm should be trivial. the the the algorithm should be trivial. And you're cuz I've I've paid $2 for And you're cuz I've I've paid $2 for And you're cuz I've I've paid $2 for apps on the phone to do this and you apps on the phone to do this and you apps on the phone to do this and you think that this is think that this is think that this is frontier level. Well, I've told you why frontier level. Well, I've told you why frontier level. Well, I've told you why the phone the phone the phone app is is a much simpler problem. app is is a much simpler problem. app is is a much simpler problem. >> So what is the problem and why is the >> So what is the problem and why is the >> So what is the problem and why is the phone app better? So this this is phone app better? So this this is phone app better? So this this is actually actually actually right it's beyond the frontier of right it's beyond the frontier of right it's beyond the frontier of today's model. today's model. today's model. >> I love that he's steepled his fingers. >> I love that he's steepled his fingers. >> I love that he's steepled his fingers. Let me tell you why this is hard. Let me tell you why this is hard. Let me tell you why this is hard. >> [laughter] >> So panorama stitching. So the idea is >> So panorama stitching. So the idea is I want I want I want to you know, ZoomIt which lets you to you know, ZoomIt which lets you to you know, ZoomIt which lets you capture screen snippets capture screen snippets capture screen snippets to be able to capture a screen snippet to be able to capture a screen snippet to be able to capture a screen snippet that gets stitched into a uh that gets stitched into a uh that gets stitched into a uh extended image. And the way you would do extended image. And the way you would do extended image. And the way you would do it is select an area of the screen and it is select an area of the screen and it is select an area of the screen and then scroll then scroll then scroll the content through that the content through that the content through that window through that portal.

window through that portal. window through that portal. And what the tool would do is go And what the tool would do is go And what the tool would do is go say you scrolled for example your inbox, say you scrolled for example your inbox, say you scrolled for example your inbox, you know, it doesn't all fit on one you know, it doesn't all fit on one you know, it doesn't all fit on one screen you select your inbox uh screen you select your inbox uh screen you select your inbox uh in ZoomIt and then you scroll and more in ZoomIt and then you scroll and more in ZoomIt and then you scroll and more emails are coming through the scroll emails are coming through the scroll emails are coming through the scroll window. window. window. >> you know, we we may we wrote a seven, >> you know, we we may we wrote a seven, >> you know, we we may we wrote a seven, eight page paper, we go to that website, eight page paper, we go to that website, eight page paper, we go to that website, we select a window, a viewport and then we select a window, a viewport and then we select a window, a viewport and then you scroll through it and then you're you scroll through it and then you're you scroll through it and then you're going screenshot, screenshot, going screenshot, screenshot, going screenshot, screenshot, screenshot, screenshot, screenshot. screenshot, screenshot, screenshot. screenshot, screenshot, screenshot. >> And you're doing >> And you're doing >> And you're doing non-JPEG, you're doing PNGs, lossless. non-JPEG, you're doing PNGs, lossless. non-JPEG, you're doing PNGs, lossless. Yeah. And then Yeah. And then Yeah. And then what you want is all of that to be what you want is all of that to be what you want is all of that to be stitched into one stitched into one stitched into one long image that is shows you all of the long image that is shows you all of the long image that is shows you all of the full document. With high fidelity. Yeah. full document. With high fidelity. Yeah. full document. With high fidelity. Yeah. Or do it horizontally. Right, right, Or do it horizontally. Right, right, Or do it horizontally. Right, right, right. right. right. >> So the challenge is that as you if you >> So the challenge is that as you if you >> So the challenge is that as you if you let the user scroll let the user scroll let the user scroll and you're just and you're just and you're just periodically as as a timer fires periodically as as a timer fires periodically as as a timer fires grabbing images, you have no idea how grabbing images, you have no idea how grabbing images, you have no idea how much they've scrolled. Right. much they've scrolled. Right. much they've scrolled. Right. You just know that this image came a few You just know that this image came a few You just know that this image came a few seconds after the other one. So you seconds after the other one. So you seconds after the other one. So you don't know the offset between one image don't know the offset between one image don't know the offset between one image and the other. It could have been no and the other. It could have been no and the other. It could have been no overlap no no 100% overlap.

overlap no no 100% overlap. overlap no no 100% overlap. >> they could move 20 pixels or a thousand >> they could move 20 pixels or a thousand >> they could move 20 pixels or a thousand between between between >> Or or you might have accidentally >> Or or you might have accidentally >> Or or you might have accidentally scrolled backwards. scrolled backwards. scrolled backwards. You know, twitched and scrolled You know, twitched and scrolled You know, twitched and scrolled backwards. Okay. Not And so, you know, backwards. Okay. Not And so, you know, backwards. Okay. Not And so, you know, the obvious thing is, "Oh, well, you the obvious thing is, "Oh, well, you the obvious thing is, "Oh, well, you know what? You know, the way you deal know what? You know, the way you deal know what? You know, the way you deal with that is just with that is just with that is just doing bit split compares, you know, bit doing bit split compares, you know, bit doing bit split compares, you know, bit memory compares and find where things memory compares and find where things memory compares and find where things line up and then go, "Oh, that's that's line up and then go, "Oh, that's that's line up and then go, "Oh, that's that's what it is." Well, two problems with what it is." Well, two problems with what it is." Well, two problems with that. One of them is extremely that. One of them is extremely that. One of them is extremely inefficient. You know, if you've got inefficient. You know, if you've got inefficient. You know, if you've got uh large a high-res monitor and you uh large a high-res monitor and you uh large a high-res monitor and you capture your inbox which is the whole capture your inbox which is the whole capture your inbox which is the whole vertical height you know, you've got vertical height you know, you've got vertical height you know, you've got over a thousand pixels there. over a thousand pixels there. over a thousand pixels there. Uh with whatever width that you're now Uh with whatever width that you're now Uh with whatever width that you're now having for every one of these grabbed having for every one of these grabbed having for every one of these grabbed images having to do pixel by pixel images having to do pixel by pixel images having to do pixel by pixel comparisons to see where it lines up for comparisons to see where it lines up for comparisons to see where it lines up for every single image in the scroll. So way every single image in the scroll. So way every single image in the scroll. So way of way too inefficient. The second of way too inefficient. The second of way too inefficient. The second problem problem problem is that you might not even be able to is that you might not even be able to is that you might not even be able to line them up right. line them up right. line them up right. Even doing that because they're not Even doing that because they're not Even doing that because they're not going to be pixel perfect matches.

going to be pixel perfect matches. going to be pixel perfect matches. If you've got something like ClearType If you've got something like ClearType If you've got something like ClearType on on on as you shift even vertically ClearType as you shift even vertically ClearType as you shift even vertically ClearType changes the colors of pixels based on changes the colors of pixels based on changes the colors of pixels based on this algorithm which is kind of this algorithm which is kind of this algorithm which is kind of non-deterministic about how it's going non-deterministic about how it's going non-deterministic about how it's going to do the edges. And so to do the edges. And so to do the edges. And so a letter A as it shifts pixel by pixel a letter A as it shifts pixel by pixel a letter A as it shifts pixel by pixel has the edges of it other pixel colors has the edges of it other pixel colors has the edges of it other pixel colors beyond the text color shift as well and beyond the text color shift as well and beyond the text color shift as well and so you'll never find the match. so you'll never find the match. so you'll never find the match. And so you've got to use these And so you've got to use these And so you've got to use these heuristics both to try to figure out how heuristics both to try to figure out how heuristics both to try to figure out how much did they scroll? Did they scroll much did they scroll? Did they scroll much did they scroll? Did they scroll accidentally up? Did they scroll down? accidentally up? Did they scroll down? accidentally up? Did they scroll down? Try to guess exactly how much they might Try to guess exactly how much they might Try to guess exactly how much they might have scrolled, start your search there have scrolled, start your search there have scrolled, start your search there and then the search has to be a and then the search has to be a and then the search has to be a coarse-grained search to see, "Hey, does coarse-grained search to see, "Hey, does coarse-grained search to see, "Hey, does this look like it'll line up?" And then this look like it'll line up?" And then this look like it'll line up?" And then let's take say good level check and get let's take say good level check and get let's take say good level check and get some confidence. some confidence. some confidence. But most of a lot of times you can never But most of a lot of times you can never But most of a lot of times you can never be 100% certain. be 100% certain. be 100% certain. And so And so And so with all that complexity the trying to with all that complexity the trying to with all that complexity the trying to be efficient and trying to find a match be efficient and trying to find a match be efficient and trying to find a match and trying to guess and trying to guess and trying to guess at the same time not having that at the same time not having that at the same time not having that information that would have been really information that would have been really information that would have been really helpful makes it a really really really helpful makes it a really really really helpful makes it a really really really hard problem and lots with lots of edge hard problem and lots with lots of edge hard problem and lots with lots of edge cases by the way. Yeah. Yeah. And this cases by the way. Yeah. Yeah. And this cases by the way. Yeah. Yeah. And this the edge cases part I think is what's the edge cases part I think is what's the edge cases part I think is what's fun because you told me this was hard fun because you told me this was hard fun because you told me this was hard and I have a $1.99 app on the phone that and I have a $1.99 app on the phone that and I have a $1.99 app on the phone that does it. But to your point is it's it's does it. But to your point is it's it's does it. But to your point is it's it's very constrained in what it's doing. It very constrained in what it's doing. It very constrained in what it's doing. It knows the size of the screen. It And it knows the size of the screen. It And it knows the size of the screen. It And it knows exactly how much assumptions. Like knows exactly how much assumptions. Like knows exactly how much assumptions. Like if you if the app could control the if you if the app could control the if you if the app could control the scroll and know, "Oh, I scrolled 50 scroll and know, "Oh, I scrolled 50 scroll and know, "Oh, I scrolled 50 pixels." That would be easy. Yeah. And pixels." That would be easy. Yeah. And pixels." That would be easy. Yeah. And that's what you get with the phone is

that's what you get with the phone is that's what you get with the phone is like when you swipe it knows exactly how like when you swipe it knows exactly how like when you swipe it knows exactly how much swiped. You know, the much swiped. You know, the much swiped. You know, the the it can look at the scroll. It's and the it can look at the scroll. It's and the it can look at the scroll. It's and [clears throat] so if you have the [clears throat] so if you have the [clears throat] so if you have the scroll distance then it's a trivial scroll distance then it's a trivial scroll distance then it's a trivial problem. problem. problem. >> Yeah. Now the one thing that's just then >> Yeah. Now the one thing that's just then >> Yeah. Now the one thing that's just then it's really hard. it's really hard. it's really hard. One of One of that people forget of a One of One of that people forget of a One of One of that people forget of a certain age is that we were of a of the certain age is that we were of a of the certain age is that we were of a of the age when we could actually see pixels. age when we could actually see pixels. age when we could actually see pixels. Like a single dot was visible. And now Like a single dot was visible. And now Like a single dot was visible. And now we're in retina level you know DPI. Over we're in retina level you know DPI. Over we're in retina level you know DPI. Over 100, 200 DPI you can't tell. 100, 200 DPI you can't tell. 100, 200 DPI you can't tell. Back in the day RGB red, green, blue Back in the day RGB red, green, blue Back in the day RGB red, green, blue pixels were you know there was sub pixel pixels were you know there was sub pixel pixels were you know there was sub pixel addressing and Bill Hilf and the folks addressing and Bill Hilf and the folks addressing and Bill Hilf and the folks that did TrueType when going left to that did TrueType when going left to that did TrueType when going left to right you would have red, green right you would have red, green right you would have red, green What's zoom in? You can zoom in down to What's zoom in? You can zoom in down to What's zoom in? You can zoom in down to the the the Actually we should do that. We have a Actually we should do that. We have a Actually we should do that. We have a great point. Let me let me zoom in on great point. Let me let me zoom in on great point. Let me let me zoom in on something stupid. something stupid. something stupid. I'll share my screen here and we'll zoom I'll share my screen here and we'll zoom I'll share my screen here and we'll zoom in and we'll take a look at the sub in and we'll take a look at the sub in and we'll take a look at the sub pixel addressing. So that looks like a you know light gray So that looks like a you know light gray text but you can see blue lines light text but you can see blue lines light text but you can see blue lines light light blue and dark blue light blue and dark blue light blue and dark blue around and then orange and red too.

around and then orange and red too. around and then orange and red too. Yeah. And there and and all the things Yeah. And there and and all the things Yeah. And there and and all the things in between which is really really in between which is really really in between which is really really interesting. Look at this G over here. interesting. Look at this G over here. interesting. Look at this G over here. There's a lot going on there. So what There's a lot going on there. So what There's a lot going on there. So what you think is gray you think is gray you think is gray is not gray at all. And even the blue text has red there. And even the blue text has red there. And then Windows has a thing that no one And then Windows has a thing that no one And then Windows has a thing that no one ever uses. ever uses. ever uses. I don't know except for you the clear I don't know except for you the clear I don't know except for you the clear clear type tuner. If you type clear type clear type tuner. If you type clear type clear type tuner. If you type clear type >> Never use that. I didn't even know there >> Never use that. I didn't even know there >> Never use that. I didn't even know there was such a thing. was such a thing. was such a thing. >> know about this? No. Oh brother. Watch >> know about this? No. Oh brother. Watch >> know about this? No. Oh brother. Watch this. this. this. So you pick your monitors. We'll tune So you pick your monitors. We'll tune So you pick your monitors. We'll tune we'll tune this one here. Just the one. we'll tune this one here. Just the one. we'll tune this one here. Just the one. Okay. Now when you tune the text on the Okay. Now when you tune the text on the Okay. Now when you tune the text on the monitor you can pick which monitor monitor you can pick which monitor monitor you can pick which monitor is the right one and then you pick which is the right one and then you pick which is the right one and then you pick which text sample looks text sample looks text sample looks >> Oh I remember that. I Do you know when I >> Oh I remember that. I Do you know when I >> Oh I remember that. I Do you know when I use this? When they introduced this in use this? When they introduced this in use this? When they introduced this in Windows Vista. Yeah yeah yeah. Yeah. So Windows Vista. Yeah yeah yeah. Yeah. So Windows Vista. Yeah yeah yeah. Yeah. So you see we've got you see we've got you see we've got a a a a single pixel here is red, green, blue a single pixel here is red, green, blue a single pixel here is red, green, blue right? And I zoom out and they're asking right? And I zoom out and they're asking right? And I zoom out and they're asking me do I want it to look like this with a me do I want it to look like this with a me do I want it to look like this with a light blue? Look at the I and sit. Yeah.

light blue? Look at the I and sit. Yeah. light blue? Look at the I and sit. Yeah. And then we'll go over to here. And then we'll go over to here. And then we'll go over to here. Oh it's Oh it's Oh it's it's actually flipped. Look the blue's it's actually flipped. Look the blue's it's actually flipped. Look the blue's on the left and the right. Because some on the left and the right. Because some on the left and the right. Because some monitors are not red, green, blue monitors are not red, green, blue monitors are not red, green, blue they're blue, green, red. Yeah. they're blue, green, red. Yeah. they're blue, green, red. Yeah. Yeah. So you can pick the one that makes Yeah. So you can pick the one that makes Yeah. So you can pick the one that makes you happy and go through it and then you you happy and go through it and then you you happy and go through it and then you get into these situations where it's get into these situations where it's get into these situations where it's like which one of these looks better to like which one of these looks better to like which one of these looks better to your eye? your eye? your eye? If we go back over to here If we go back over to here If we go back over to here See how dark the blue is on the I and See how dark the blue is on the I and See how dark the blue is on the I and sit? And then if we go over here sit? And then if we go over here sit? And then if we go over here it's much lighter. it's much lighter. it's much lighter. Yeah. So when you're adjusting the Yeah. So when you're adjusting the Yeah. So when you're adjusting the relative darkness of something in relative darkness of something in relative darkness of something in TrueType and this is built into Windows TrueType and this is built into Windows TrueType and this is built into Windows you can do this anytime. You're actually you can do this anytime. You're actually you can do this anytime. You're actually picking how dark the red and the blue picking how dark the red and the blue picking how dark the red and the blue are and whether or not it like you like are and whether or not it like you like are and whether or not it like you like the vibe. That's called the clear type the vibe. That's called the clear type the vibe. That's called the clear type tuner. tuner. tuner. And to your point now now when I'm And to your point now now when I'm And to your point now now when I'm trivializing what I like to do is is in trivializing what I like to do is is in trivializing what I like to do is is in a general rule is Mark will tell me a general rule is Mark will tell me a general rule is Mark will tell me something and I'll just trivialize it. something and I'll just trivialize it. something and I'll just trivialize it. Yeah. [snorts] Yeah. [snorts] Yeah. [snorts] Cuz that's the way your brain works. Cuz that's the way your brain works. Cuz that's the way your brain works. Well cuz this over simplification is Well cuz this over simplification is Well cuz this over simplification is making It's all I can handle. making It's all I can handle. making It's all I can handle. Yeah. It's pretty much I'm pretty sure Yeah. It's pretty much I'm pretty sure Yeah. It's pretty much I'm pretty sure you can do that with a for loop there you can do that with a for loop there you can do that with a for loop there Mark.

Mark. Mark. So I said you were talking about the So I said you were talking about the So I said you were talking about the red, green so I solved that problem by red, green so I solved that problem by red, green so I solved that problem by cuz I tried to one shot cuz Mark tells cuz I tried to one shot cuz Mark tells cuz I tried to one shot cuz Mark tells me he's working on something for 60 me he's working on something for 60 me he's working on something for 60 hours and I try to do it in one shot. Um hours and I try to do it in one shot. Um hours and I try to do it in one shot. Um So I took the you know thousand by a So I took the you know thousand by a So I took the you know thousand by a thousand thing. I converted it to gray thousand thing. I converted it to gray thousand thing. I converted it to gray scale which cuts out all of that color scale which cuts out all of that color scale which cuts out all of that color and then I downscaled it to one quarter and then I downscaled it to one quarter and then I downscaled it to one quarter res. res. res. And then I focused on the region most And then I focused on the region most And then I focused on the region most likely that overlap. I take the top 10 likely that overlap. I take the top 10 likely that overlap. I take the top 10 20% and the bottom 10 20% and then just 20% and the bottom 10 20% and then just 20% and the bottom 10 20% and then just kind of do a phase correlation between kind of do a phase correlation between kind of do a phase correlation between them and then if I find it I pull out to them and then if I find it I pull out to them and then if I find it I pull out to full res and then just do a little local full res and then just do a little local full res and then just do a little local search adjustment. And that worked search adjustment. And that worked search adjustment. And that worked really really fast really really fast really really fast until it didn't work at all. Yeah and in until it didn't work at all. Yeah and in until it didn't work at all. Yeah and in fact some one of the edge cases that I fact some one of the edge cases that I fact some one of the edge cases that I was struggling to make both efficient was struggling to make both efficient was struggling to make both efficient and accurate Mhm. is VS code editor page and accurate Mhm. is VS code editor page and accurate Mhm. is VS code editor page where you just pressed enter and so you where you just pressed enter and so you where you just pressed enter and so you had just have line numbers down the had just have line numbers down the had just have line numbers down the left. So it's and the view portal is left. So it's and the view portal is left. So it's and the view portal is very wide very wide very wide very narrowly tall so only a few of the very narrowly tall so only a few of the very narrowly tall so only a few of the numbers line numbers are on the left numbers line numbers are on the left numbers line numbers are on the left side of it yeah.

side of it yeah. side of it yeah. And then you scroll through it. And the And then you scroll through it. And the And then you scroll through it. And the problem there is that there's almost no problem there is that there's almost no problem there is that there's almost no difference between difference between difference between one image and one and the next one one image and one and the next one one image and one and the next one that's that's shifted. In fact it's like that's that's shifted. In fact it's like that's that's shifted. In fact it's like a fraction of a percent of the pixels a fraction of a percent of the pixels a fraction of a percent of the pixels are different. Yeah. And so once you are different. Yeah. And so once you are different. Yeah. And so once you start to do the optimizations of I'm start to do the optimizations of I'm start to do the optimizations of I'm just going to go coarse grain it's like just going to go coarse grain it's like just going to go coarse grain it's like well these are all the same image or well these are all the same image or well these are all the same image or Yeah. Yeah. Yeah. Or and then if you do coarse grain Or and then if you do coarse grain Or and then if you do coarse grain matching of where things go and overlap matching of where things go and overlap matching of where things go and overlap then and again you have the clear type then and again you have the clear type then and again you have the clear type problem on top of this. So it's problem on top of this. So it's problem on top of this. So it's >> Well but the clear type problem goes >> Well but the clear type problem goes >> Well but the clear type problem goes away if you gray scale it. away if you gray scale it. away if you gray scale it. Not not totally. Not not totally cuz Not not totally. Not not totally cuz Not not totally. Not not totally cuz cuz cuz cuz you could still get color differences in you could still get color differences in you could still get color differences in uh Okay. roughness um uh Okay. roughness um uh Okay. roughness um So then I would I I I was thinking well So then I would I I I was thinking well So then I would I I I was thinking well maybe we would have like a CRC of some maybe we would have like a CRC of some maybe we would have like a CRC of some kind of like this chunk. I would chunk kind of like this chunk. I would chunk kind of like this chunk. I would chunk the thing up into into into stripes the thing up into into into stripes the thing up into into into stripes and then uh and then uh and then uh come up with a come up with a come up with a a hash of some kind to express a hash of some kind to express a hash of some kind to express the the the uniqueness of that of that the the the uniqueness of that of that the the the uniqueness of that of that strip and then I would test the strips strip and then I would test the strips strip and then I would test the strips and then you had a situation where and then you had a situation where and then you had a situation where you gave me like a couple of hundred you gave me like a couple of hundred you gave me like a couple of hundred PNGs and I saw them shift and then it PNGs and I saw them shift and then it PNGs and I saw them shift and then it was like these only move like two or was like these only move like two or was like these only move like two or three. So I skipped over and I'm like three. So I skipped over and I'm like three. So I skipped over and I'm like I'm going to pick this one skip these I'm going to pick this one skip these I'm going to pick this one skip these two or three and pick the one that's two or three and pick the one that's two or three and pick the one that's like a proper scroll down and then you like a proper scroll down and then you like a proper scroll down and then you did end up doing that as well throwing did end up doing that as well throwing did end up doing that as well throwing out the small moves and that seemed to out the small moves and that seemed to out the small moves and that seemed to help as well. Let me give you another help as well. Let me give you another help as well. Let me give you another one. Another problem is that you've got

one. Another problem is that you've got one. Another problem is that you've got if you've got repetitive content though if you've got repetitive content though if you've got repetitive content though too you could end up matching on the too you could end up matching on the too you could end up matching on the wrong wrong wrong uh part of it as well. I mean it's just uh part of it as well. I mean it's just uh part of it as well. I mean it's just like like like I'm you know I thought I was pretty much I'm you know I thought I was pretty much I'm you know I thought I was pretty much robust yesterday morning. robust yesterday morning. robust yesterday morning. You told me you were done. I know. You told me you were done. I know. You told me you were done. I know. [clears throat] [clears throat] [clears throat] >> me you were done. Well then I you know >> me you were done. Well then I you know >> me you were done. Well then I you know I'm like trying my own pianos all over I'm like trying my own pianos all over I'm like trying my own pianos all over the place with different settings and the place with different settings and the place with different settings and then I came across another edge then I came across another edge then I came across another edge condition. And now I'm down a rabbit condition. And now I'm down a rabbit condition. And now I'm down a rabbit hole where trying to fix that is hole where trying to fix that is hole where trying to fix that is regressing other things. And I'm just it's just a my nightmare. I And I'm just it's just a my nightmare. I mean I mean I mean I as you know I'm a perfectionist and one as you know I'm a perfectionist and one as you know I'm a perfectionist and one of the things that that it's either a of the things that that it's either a of the things that that it's either a good it's probably a good trait at at good it's probably a good trait at at good it's probably a good trait at at large but for something like this large but for something like this large but for something like this I've spent so much time on trying to get I've spent so much time on trying to get I've spent so much time on trying to get this right. Yeah. You've nerd sniped this right. Yeah. You've nerd sniped this right. Yeah. You've nerd sniped yourself. yourself. yourself. >> And I nerd sniped myself and the problem >> And I nerd sniped myself and the problem >> And I nerd sniped myself and the problem is like I said there's two issues here. is like I said there's two issues here. is like I said there's two issues here. AI can't see the pixels Yeah. very AI can't see the pixels Yeah. very AI can't see the pixels Yeah. very clearly. clearly. clearly. And so it's kind of part it kind of sees And so it's kind of part it kind of sees And so it's kind of part it kind of sees them you know cuz it's it understands them you know cuz it's it understands them you know cuz it's it understands images but it can't see the subtle images but it can't see the subtle images but it can't see the subtle errors in the stitches.

errors in the stitches. errors in the stitches. And so I've got so I'm now helping it And so I've got so I'm now helping it And so I've got so I'm now helping it it can't it's not fully automated. So a it can't it's not fully automated. So a it can't it's not fully automated. So a combination of this is really really combination of this is really really combination of this is really really hard lots of constraints hard lots of constraints hard lots of constraints and lots of heuristics that all have to and lots of heuristics that all have to and lots of heuristics that all have to be made at the same time for all the be made at the same time for all the be made at the same time for all the different cases. Yeah. And I've got it different cases. Yeah. And I've got it different cases. Yeah. And I've got it by the way with synthetic tests where it by the way with synthetic tests where it by the way with synthetic tests where it that are stressing the cases and when that are stressing the cases and when that are stressing the cases and when there's a failure I have it add another there's a failure I have it add another there's a failure I have it add another test and try to so I try to try to test and try to so I try to try to test and try to so I try to try to prevent regression of that. It is it is prevent regression of that. It is it is prevent regression of that. It is it is it is one of those things that I think it is one of those things that I think it is one of those things that I think would be a great senior project. Oh would be a great senior project. Oh would be a great senior project. Oh yeah. You can get to 80% fast. I got to yeah. You can get to 80% fast. I got to yeah. You can get to 80% fast. I got to 80% in a couple of hours of Python 80% in a couple of hours of Python 80% in a couple of hours of Python >> [clears throat] >> [clears throat] >> [clears throat] >> and then I and then you kept throwing >> and then I and then you kept throwing >> and then I and then you kept throwing edge cases at me and I was like that's edge cases at me and I was like that's edge cases at me and I was like that's fine. fine. fine. And I was like I'm done and I put mine And I was like I'm done and I put mine And I was like I'm done and I put mine up on my you can see mine up on my on my up on my you can see mine up on my on my up on my you can see mine up on my on my thing and it'll work if you squint thing and it'll work if you squint thing and it'll work if you squint 80% of the time which is a B a low B. 80% of the time which is a B a low B. 80% of the time which is a B a low B. Yeah. Yeah. Yeah. >> [laughter] >> [laughter] >> [laughter] >> And Mark's like no it must be perfect. >> And Mark's like no it must be perfect. >> And Mark's like no it must be perfect. So are you going to sell it for $4.99 So are you going to sell it for $4.99 So are you going to sell it for $4.99 and or is this going to be like a B? and or is this going to be like a B? and or is this going to be like a B? It's actually in the next release of It's actually in the next release of It's actually in the next release of Emit. Emit. Emit. Oh really? You're giving it to the Oh really? You're giving it to the Oh really? You're giving it to the people? Giving it to the people and and people? Giving it to the people and and people? Giving it to the people and and open source too.

open source too. open source too. But that's awesome. But that's awesome. But that's awesome. >> Interesting thing about this is is it >> Interesting thing about this is is it >> Interesting thing about this is is it this really is just beyond I don't know this really is just beyond I don't know this really is just beyond I don't know how far beyond but it's definitely how far beyond but it's definitely how far beyond but it's definitely beyond the capabilities of today's AI. beyond the capabilities of today's AI. beyond the capabilities of today's AI. Yeah. Totally. Okay so Yeah. Totally. Okay so Yeah. Totally. Okay so we've talked before about the the we've talked before about the the we've talked before about the the metaphor of a million monkeys with a metaphor of a million monkeys with a metaphor of a million monkeys with a million typewriters and if a million million typewriters and if a million million typewriters and if a million monkeys and a million typewriters or an monkeys and a million typewriters or an monkeys and a million typewriters or an infinite number of monkeys and infinite number of monkeys and infinite number of monkeys and typewriters could go and do this you typewriters could go and do this you typewriters could go and do this you don't think without the right harness don't think without the right harness don't think without the right harness around it they could ever solve this around it they could ever solve this around it they could ever solve this problem. Well actually let me give So problem. Well actually let me give So problem. Well actually let me give So there's a few things that I ran into and there's a few things that I ran into and there's a few things that I ran into and I'm sure you ran into this too. I'm sure you ran into this too. I'm sure you ran into this too. It's as as um It's as as um It's as as um it would do its own kind of tests and it would do its own kind of tests and it would do its own kind of tests and say it passes and then I'd point out say it passes and then I'd point out say it passes and then I'd point out something else failed and it would then something else failed and it would then something else failed and it would then try to understand what that failure is try to understand what that failure is try to understand what that failure is and then make an update and then it and then make an update and then it and then make an update and then it would fix that but not even run the would fix that but not even run the would fix that but not even run the other tests and then it it would run all other tests and then it it would run all other tests and then it it would run all the other tests the other tests the other tests it would um it would um it would um you know find a regression and then have you know find a regression and then have you know find a regression and then have to go and um to go and um to go and um and and juggle it but and and juggle it but and and juggle it but one of the things that one of the things that one of the things that it wasn't doing until I told it cuz it'd it wasn't doing until I told it cuz it'd it wasn't doing until I told it cuz it'd be I'd be like okay that worked. It be I'd be like okay that worked. It be I'd be like okay that worked. It wouldn't automatically try to wouldn't automatically try to wouldn't automatically try to add a stress test case add a stress test case add a stress test case to capture that failure so that you know to capture that failure so that you know to capture that failure so that you know it couldn't regress it.

it couldn't regress it. it couldn't regress it. I had to tell it and guide it and every I had to tell it and guide it and every I had to tell it and guide it and every time it there's a failure I'm like add time it there's a failure I'm like add time it there's a failure I'm like add another case to the stress test another case to the stress test another case to the stress test that targets that. that targets that. that targets that. Ensure that it fails on the current Ensure that it fails on the current Ensure that it fails on the current version. version. version. Fix the problem. Ensure that that stress Fix the problem. Ensure that that stress Fix the problem. Ensure that that stress test pass and then only then run the test pass and then only then run the test pass and then only then run the full stress test cuz the full stress full stress test cuz the full stress full stress test cuz the full stress test now takes you know a minute to run. test now takes you know a minute to run. test now takes you know a minute to run. Yeah. This is an example of how your Yeah. This is an example of how your Yeah. This is an example of how your cost of cost of cost of coding goes to can go to zero and maybe coding goes to can go to zero and maybe coding goes to can go to zero and maybe it will but the cost of testing is just it will but the cost of testing is just it will but the cost of testing is just like physics. like physics. like physics. And so a few you know interesting things And so a few you know interesting things And so a few you know interesting things that really highlight the edges of how that really highlight the edges of how that really highlight the edges of how far this stuff's going to get us. Yeah. far this stuff's going to get us. Yeah. far this stuff's going to get us. Yeah. I keep I I know that it's cliche but I I keep I I know that it's cliche but I I keep I I know that it's cliche but I keep coming back to the 80/20 rule. It keep coming back to the 80/20 rule. It keep coming back to the 80/20 rule. It You get 80% of the 20% of the work and You get 80% of the 20% of the work and You get 80% of the 20% of the work and you're going to spend a bunch of time you're going to spend a bunch of time you're going to spend a bunch of time asymptotically approaching 100 and asymptotically approaching 100 and asymptotically approaching 100 and you're never going to get there. By the you're never going to get there. By the you're never going to get there. By the way is traditional software you know way is traditional software you know way is traditional software you know coding. Seems familiar doesn't it? Yeah coding. Seems familiar doesn't it? Yeah coding. Seems familiar doesn't it? Yeah it does. It's like an always like a It's it does. It's like an always like a It's it does. It's like an always like a It's like, "Look, I'm almost done. We can like, "Look, I'm almost done. We can like, "Look, I'm almost done. We can ship this tomorrow." This is why the ship this tomorrow." This is why the ship this tomorrow." This is why the progress bar in Windows is still at 99% progress bar in Windows is still at 99% progress bar in Windows is still at 99% and just like and just like and just like No, it's not It's not done yet. Yeah.

No, it's not It's not done yet. Yeah. No, it's not It's not done yet. Yeah. Cool. Cool. Cool. Um Um Um Uh uh that was a fun show. I liked this Uh uh that was a fun show. I liked this Uh uh that was a fun show. I liked this show. That was a good episode. I liked show. That was a good episode. I liked show. That was a good episode. I liked it. it. it. Yeah? I think we'll call that a show. Yeah? I think we'll call that a show. Yeah? I think we'll call that a show. There's two shows there, but I don't There's two shows there, but I don't There's two shows there, but I don't know what to call them, but we learned know what to call them, but we learned know what to call them, but we learned something. Mark and Scott learned to something. Mark and Scott learned to something. Mark and Scott learned to stitch a panorama. stitch a panorama. stitch a panorama. Uh Uh Uh and uh and uh and uh I don't know. Align a model. Uh I don't know. Align a model. Uh I don't know. Align a model. Uh yeah. What did you learn? yeah. What did you learn? yeah. What did you learn? I learned that you're a perfectionist. I learned that you're a perfectionist. I learned that you're a perfectionist. You already knew that though, didn't You already knew that though, didn't You already knew that though, didn't you? you? you? >> And you learned that I apparently am >> And you learned that I apparently am >> And you learned that I apparently am very lazy. very lazy. very lazy. I already knew that too. I already knew that too. I already knew that too. >> [laughter] >> All right. If you made it to this far, >> All right. If you made it to this far, if you made it to the end of this if you made it to the end of this if you made it to the end of this episode, please tell a friend, like and episode, please tell a friend, like and episode, please tell a friend, like and subscribe, and uh leave a comment to let subscribe, and uh leave a comment to let subscribe, and uh leave a comment to let us know you made it this far. We us know you made it this far. We us know you made it this far. We appreciate you all, and also let us know appreciate you all, and also let us know appreciate you all, and also let us know where you saw this, because we're going where you saw this, because we're going where you saw this, because we're going to try to put it out on the YouTube. Uh to try to put it out on the YouTube. Uh to try to put it out on the YouTube. Uh we might put Actually, you know, we we might put Actually, you know, we we might put Actually, you know, we should put some pieces of this on the should put some pieces of this on the should put some pieces of this on the TikTok. The kids love the talk. And also TikTok. The kids love the talk. And also TikTok. The kids love the talk. And also uh put in the comments any topics you uh put in the comments any topics you uh put in the comments any topics you want us to discuss, technical, want us to discuss, technical, want us to discuss, technical, non-technical, That's a great point. non-technical, That's a great point. non-technical, That's a great point. Yeah. What uh what topic should we Yeah. What uh what topic should we Yeah. What uh what topic should we learn? What should Mark and Scott learn learn? What should Mark and Scott learn learn? What should Mark and Scott learn to next time to next time to next time on the Mark and Scott Scott and Mark on the Mark and Scott Scott and Mark on the Mark and Scott Scott and Mark podcast.