SKY ENGINE AI's Dr. Malc Souter on AI-Generated Training Data

Hi, I'm Scott Hansselman. This is Hi, I'm Scott Hansselman. This is another episode of Hansel Minutes. Today another episode of Hansel Minutes. Today another episode of Hansel Minutes. Today I have the pleasure of chatting with Dr. I have the pleasure of chatting with Dr. I have the pleasure of chatting with Dr. Malc Sudter. He's got a PhD in computer Malc Sudter. He's got a PhD in computer Malc Sudter. He's got a PhD in computer graphics algorithms and is currently at graphics algorithms and is currently at graphics algorithms and is currently at Sky Engine AI. How are you, sir? Sky Engine AI. How are you, sir? Sky Engine AI. How are you, sir? I'm very good. Very happy to be on your I'm very good. Very happy to be on your I'm very good. Very happy to be on your podcast. Thank you so much for hanging podcast. Thank you so much for hanging podcast. Thank you so much for hanging out with me today. I I have to say I've out with me today. I I have to say I've out with me today. I I have to say I've said it before on the show, but I'm said it before on the show, but I'm said it before on the show, but I'm always so impressed and a little bit always so impressed and a little bit always so impressed and a little bit intimidated with uh with people who have intimidated with uh with people who have intimidated with uh with people who have PhDs in a very specific place. It's just PhDs in a very specific place. It's just PhDs in a very specific place. It's just um I think it's because it's you have an um I think it's because it's you have an um I think it's because it's you have an expertise in something that's so expertise in something that's so expertise in something that's so specific that you must have had passion specific that you must have had passion specific that you must have had passion for it. Did you always know you wanted for it. Did you always know you wanted for it. Did you always know you wanted to dig into computer graphics to the to dig into computer graphics to the to dig into computer graphics to the point that you would do graduate and point that you would do graduate and point that you would do graduate and post-graduate work? post-graduate work? post-graduate work? Well, yes. And how it happened was I was Well, yes. And how it happened was I was Well, yes. And how it happened was I was doing the wrong degree, which was doing the wrong degree, which was doing the wrong degree, which was electronic engineering. And halfway electronic engineering. And halfway electronic engineering. And halfway through that, a great lecturer, Lads and through that, a great lecturer, Lads and through that, a great lecturer, Lads and Hayes from South Africa came came Hayes from South Africa came came Hayes from South Africa came came bounding in and said, "I'm going to give bounding in and said, "I'm going to give bounding in and said, "I'm going to give you handouts today, and I'm going to you handouts today, and I'm going to you handouts today, and I'm going to show you these um these animations, show you these um these animations, show you these um these animations, computer graphics animation from computer graphics animation from computer graphics animation from Sigraph, which is like the big academic Sigraph, which is like the big academic Sigraph, which is like the big academic um conference for computer graphics."

um conference for computer graphics." um conference for computer graphics." and he put them on. As soon as I'd saw and he put them on. As soon as I'd saw and he put them on. As soon as I'd saw them, I realized that was 100% more them, I realized that was 100% more them, I realized that was 100% more interesting than the stuff that I was interesting than the stuff that I was interesting than the stuff that I was studying in the electronics degree. So, studying in the electronics degree. So, studying in the electronics degree. So, I basically I basically I basically was very lucky to be in a country where was very lucky to be in a country where was very lucky to be in a country where there was enough available funding, there was enough available funding, there was enough available funding, which I did go looking for, to get me which I did go looking for, to get me which I did go looking for, to get me through a master's in computer science, through a master's in computer science, through a master's in computer science, which shunted me across closer to which shunted me across closer to which shunted me across closer to computer graphics. And then I found a computer graphics. And then I found a computer graphics. And then I found a bery for a PhD up in Aberdine in North bery for a PhD up in Aberdine in North bery for a PhD up in Aberdine in North Scotland. And that allowed me to teach Scotland. And that allowed me to teach Scotland. And that allowed me to teach myself C++, go through a whole load of myself C++, go through a whole load of myself C++, go through a whole load of papers at that time which were coming papers at that time which were coming papers at that time which were coming out of Pixar and that allowed me to drag out of Pixar and that allowed me to drag out of Pixar and that allowed me to drag my carcass over to my carcass over to my carcass over to the computer science end of computer the computer science end of computer the computer science end of computer graphics. What work has happened in the graphics. What work has happened in the graphics. What work has happened in the last like five years that is last like five years that is last like five years that is revolutionary in the kind of visual revolutionary in the kind of visual revolutionary in the kind of visual effects and vision space? The reason I effects and vision space? The reason I effects and vision space? The reason I ask is that ask is that ask is that you think about the work that was being you think about the work that was being you think about the work that was being done with like Terminator 2 making like done with like Terminator 2 making like done with like Terminator 2 making like a liquid metal type person and those a liquid metal type person and those a liquid metal type person and those those effects those effects those effects good that you mentioned that that's that good that you mentioned that that's that good that you mentioned that that's that was that that was probably the most was that that was probably the most was that that was probably the most enjoyable step up in the ability enjoyable step up in the ability enjoyable step up in the ability capability of computer graphics that I capability of computer graphics that I capability of computer graphics that I saw in the cinema. And uh I went to see saw in the cinema. And uh I went to see saw in the cinema. And uh I went to see it twice. Of course, it wasn't as good it twice. Of course, it wasn't as good it twice. Of course, it wasn't as good second time round, but uh there was it second time round, but uh there was it second time round, but uh there was it was amazing. You know, nobody had ever was amazing. You know, nobody had ever was amazing. You know, nobody had ever seen anything like that.

seen anything like that. seen anything like that. Yeah. And it holds up today, though. Yeah. And it holds up today, though. Yeah. And it holds up today, though. That's the thing. Like that very much is That's the thing. Like that very much is That's the thing. Like that very much is believable. There's some edges. There's believable. There's some edges. There's believable. There's some edges. There's some compositing. There's some some compositing. There's some some compositing. There's some questionable, you know, uh shadows. But questionable, you know, uh shadows. But questionable, you know, uh shadows. But as a general rule, like that and the as a general rule, like that and the as a general rule, like that and the abyss and some of those uses of liquid abyss and some of those uses of liquid abyss and some of those uses of liquid really held up. And I feel like really held up. And I feel like really held up. And I feel like interesting, groundbreaking work was interesting, groundbreaking work was interesting, groundbreaking work was happening at that time. Here's my happening at that time. Here's my happening at that time. Here's my question. is interesting, question. is interesting, question. is interesting, groundbreaking, no one's ever seen that groundbreaking, no one's ever seen that groundbreaking, no one's ever seen that before. Work still to come. before. Work still to come. before. Work still to come. I think to be honest at this point the I think to be honest at this point the I think to be honest at this point the quality probably peaked maybe quality probably peaked maybe quality probably peaked maybe I don't know 10 to 15 years ago because I don't know 10 to 15 years ago because I don't know 10 to 15 years ago because the people that were working on the the people that were working on the the people that were working on the shots were people who had spent a shots were people who had spent a shots were people who had spent a ridiculous amount of time polishing and ridiculous amount of time polishing and ridiculous amount of time polishing and polishing. Now what's happening is in polishing. Now what's happening is in polishing. Now what's happening is in order to reduce budget order to reduce budget order to reduce budget the the work goes to lots of places the the work goes to lots of places the the work goes to lots of places around the world which is great for around the world which is great for around the world which is great for these people but they don't have quite these people but they don't have quite these people but they don't have quite the experience and I think the quality the experience and I think the quality the experience and I think the quality bar has gone down a bit but in terms of bar has gone down a bit but in terms of bar has gone down a bit but in terms of what we can see on screen you you can do what we can see on screen you you can do what we can see on screen you you can do anything you like these days really anything you like these days really anything you like these days really we've got fantastic looking far that was we've got fantastic looking far that was we've got fantastic looking far that was quite difficult for a while um fluid quite difficult for a while um fluid quite difficult for a while um fluid dynamics that was that was quite dynamics that was that was quite dynamics that was that was quite difficult for a while But uh to be difficult for a while But uh to be difficult for a while But uh to be honest these days um it's the quality is honest these days um it's the quality is honest these days um it's the quality is really about the experience of the really about the experience of the really about the experience of the people who are saying that's good enough people who are saying that's good enough people who are saying that's good enough and the people working on it and whether and the people working on it and whether and the people working on it and whether they have the budget to say well let's

they have the budget to say well let's they have the budget to say well let's spend another couple of days and tweak spend another couple of days and tweak spend another couple of days and tweak that. that. that. Does it frustrate you that people that Does it frustrate you that people that Does it frustrate you that people that are lay people have such strong are lay people have such strong are lay people have such strong opinions? You know, you'll hear Tik opinions? You know, you'll hear Tik opinions? You know, you'll hear Tik Tockers or YouTubers that'll tear apart Tockers or YouTubers that'll tear apart Tockers or YouTubers that'll tear apart a shot and they'll be like, "This is a shot and they'll be like, "This is a shot and they'll be like, "This is garbage." They, you know, Marvel or garbage." They, you know, Marvel or garbage." They, you know, Marvel or whoever, they'll pick a company is just whoever, they'll pick a company is just whoever, they'll pick a company is just not putting the effort in, and then not putting the effort in, and then not putting the effort in, and then they'll point to other shots. Uh, and they'll point to other shots. Uh, and they'll point to other shots. Uh, and now you've got YouTubers whose entire now you've got YouTubers whose entire now you've got YouTubers whose entire job is to recreate a shot to prove that job is to recreate a shot to prove that job is to recreate a shot to prove that they could do it on commercial hardware they could do it on commercial hardware they could do it on commercial hardware or publicly available hardware with or publicly available hardware with or publicly available hardware with plugins versus an entire professional VS plugins versus an entire professional VS plugins versus an entire professional VS VFX group. VFX group. VFX group. Yeah, I I I think it's fine that that Yeah, I I I think it's fine that that Yeah, I I I think it's fine that that goes on. I I guess it keeps the uh it goes on. I I guess it keeps the uh it goes on. I I guess it keeps the uh it pro possibly has a small effect of pro possibly has a small effect of pro possibly has a small effect of keeping the industry honest and keeping keeping the industry honest and keeping keeping the industry honest and keeping the quality level up. So I think I think the quality level up. So I think I think the quality level up. So I think I think it's absolutely fine. It will probably it's absolutely fine. It will probably it's absolutely fine. It will probably have more of an effect of a small have more of an effect of a small have more of an effect of a small increase in budget if if it has any increase in budget if if it has any increase in budget if if it has any effect of course. effect of course. effect of course. Mhm. So there's the there's the creation Mhm. So there's the there's the creation Mhm. So there's the there's the creation of graphical effects to make our eyes of graphical effects to make our eyes of graphical effects to make our eyes you know the human eye you know uh you know the human eye you know uh you know the human eye you know uh absorb and say that this is really cool absorb and say that this is really cool absorb and say that this is really cool that's really realistic but then there's that's really realistic but then there's that's really realistic but then there's also the inverse of that which is the also the inverse of that which is the also the inverse of that which is the computer perceiving something and uh the computer perceiving something and uh the computer perceiving something and uh the pipeline of like I p you know the pipeline of like I p you know the pipeline of like I p you know the computer vision pipeline on the on the computer vision pipeline on the on the computer vision pipeline on the on the way in. And I guess I'm I'm not sure I'm way in. And I guess I'm I'm not sure I'm way in. And I guess I'm I'm not sure I'm explaining explaining it correctly, but explaining explaining it correctly, but explaining explaining it correctly, but there's computers producing output and there's computers producing output and there's computers producing output and there's computers absorbing input. Are there's computers absorbing input. Are there's computers absorbing input. Are the lessons that we learn from creating

the lessons that we learn from creating the lessons that we learn from creating computer graphics something that can be computer graphics something that can be computer graphics something that can be used on the way in for the computer to used on the way in for the computer to used on the way in for the computer to perceive the world? I think I think perceive the world? I think I think perceive the world? I think I think actually it's a really good point actually it's a really good point actually it's a really good point because because because the things that we need to worry about the things that we need to worry about the things that we need to worry about for a special effect shot are the things for a special effect shot are the things for a special effect shot are the things that your eye is drawn towards that it that your eye is drawn towards that it that your eye is drawn towards that it possibly shouldn't be drawn towards or possibly shouldn't be drawn towards or possibly shouldn't be drawn towards or it should be drawn towards. So, it's all it should be drawn towards. So, it's all it should be drawn towards. So, it's all about our perception and what our brains about our perception and what our brains about our perception and what our brains tell us is important. and we're we tell us is important. and we're we tell us is important. and we're we should be homing in on the part of the should be homing in on the part of the should be homing in on the part of the image that's telling the story. Whether image that's telling the story. Whether image that's telling the story. Whether that's a specific character's face um or that's a specific character's face um or that's a specific character's face um or whether that's a part of the frame which whether that's a part of the frame which whether that's a part of the frame which is only flashing up for maybe a second is only flashing up for maybe a second is only flashing up for maybe a second or two seconds at most. You really want or two seconds at most. You really want or two seconds at most. You really want people looking in the right part of the people looking in the right part of the people looking in the right part of the of the frame at the right time and not of the frame at the right time and not of the frame at the right time and not being surprised that the actions being surprised that the actions being surprised that the actions happening across the other side of the happening across the other side of the happening across the other side of the screen. So the problem with this is that screen. So the problem with this is that screen. So the problem with this is that computers don't really care about computers don't really care about computers don't really care about that sort of thing from a human that sort of thing from a human that sort of thing from a human perspective. They need to be able to perspective. They need to be able to perspective. They need to be able to handle the the whole of the image that's handle the the whole of the image that's handle the the whole of the image that's going into them. So a computer vision going into them. So a computer vision going into them. So a computer vision system can't have these biases because system can't have these biases because system can't have these biases because it doesn't know what to throw away, it doesn't know what to throw away, it doesn't know what to throw away, what's not important to us as humans. So what's not important to us as humans. So what's not important to us as humans. So yeah, it's almost like we need to do the yeah, it's almost like we need to do the yeah, it's almost like we need to do the opposite opposite opposite and use use those lessons of things that and use use those lessons of things that and use use those lessons of things that we notice and we think are important and we notice and we think are important and we notice and we think are important and realize that they won't be important

realize that they won't be important realize that they won't be important when you come to train your computer when you come to train your computer when you come to train your computer vision vision vision model. model. model. That that just really like scratched That that just really like scratched That that just really like scratched specific neurons in my brain. I've never specific neurons in my brain. I've never specific neurons in my brain. I've never thought about it that way. You're thought about it that way. You're thought about it that way. You're absolutely right because I've been I've absolutely right because I've been I've absolutely right because I've been I've been playing with this Meta Quest. Uh, I been playing with this Meta Quest. Uh, I been playing with this Meta Quest. Uh, I got some prescription lenses that are got some prescription lenses that are got some prescription lenses that are inserted into the VR, which made my inserted into the VR, which made my inserted into the VR, which made my experience just a hundred times better. experience just a hundred times better. experience just a hundred times better. I would recommend anyone who has a I would recommend anyone who has a I would recommend anyone who has a Metaquest, pay the 50 bucks, get Metaquest, pay the 50 bucks, get Metaquest, pay the 50 bucks, get yourself prescription lenses, pop them yourself prescription lenses, pop them yourself prescription lenses, pop them in. It'll make your VR experience much in. It'll make your VR experience much in. It'll make your VR experience much better. And one of the things that's better. And one of the things that's better. And one of the things that's happening in VR is this. Don't render happening in VR is this. Don't render happening in VR is this. Don't render stuff that's out kind of the outside the stuff that's out kind of the outside the stuff that's out kind of the outside the focal view of your eyes. If you're focal view of your eyes. If you're focal view of your eyes. If you're looking over, if your gaze is here, save looking over, if your gaze is here, save looking over, if your gaze is here, save some sight CPU some sight CPU some sight CPU fiated rendering. fiated rendering. fiated rendering. Fiated rendering. Right. Yeah. Fiated rendering. Right. Yeah. Fiated rendering. Right. Yeah. So, fiated rendering is a technique by So, fiated rendering is a technique by So, fiated rendering is a technique by which you render with less fidelity the which you render with less fidelity the which you render with less fidelity the stuff that is kind of outside the focal stuff that is kind of outside the focal stuff that is kind of outside the focal view, but it also speaks to you don't view, but it also speaks to you don't view, but it also speaks to you don't know where someone's eye is. You don't know where someone's eye is. You don't know where someone's eye is. You don't know narratively where someone's eye is know narratively where someone's eye is know narratively where someone's eye is going to go. I never realized that going to go. I never realized that going to go. I never realized that computer vision is taking the frame as computer vision is taking the frame as computer vision is taking the frame as opposed to the focal point. Like the opposed to the focal point. Like the opposed to the focal point. Like the computer doesn't have an eyeball that's computer doesn't have an eyeball that's computer doesn't have an eyeball that's looking in this corner or looking in looking in this corner or looking in looking in this corner or looking in that corner. It could be. Yeah.

that corner. It could be. Yeah. that corner. It could be. Yeah. Wow. Okay. So, that that does have Wow. Okay. So, that that does have Wow. Okay. So, that that does have dramatic ramifications for looking at dramatic ramifications for looking at dramatic ramifications for looking at something for a computer to look at something for a computer to look at something for a computer to look at something and make a decision about what something and make a decision about what something and make a decision about what to do. to do. to do. Well, sure. And uh it also brings in the Well, sure. And uh it also brings in the Well, sure. And uh it also brings in the difference between creating your images difference between creating your images difference between creating your images with a gaming engine which is very much with a gaming engine which is very much with a gaming engine which is very much geared towards pleasing the human eye geared towards pleasing the human eye geared towards pleasing the human eye and doing something with a physically and doing something with a physically and doing something with a physically based renderer like we do at Sky Engine based renderer like we do at Sky Engine based renderer like we do at Sky Engine AI where we've we're basically tracing AI where we've we're basically tracing AI where we've we're basically tracing rays of light bouncing that around and rays of light bouncing that around and rays of light bouncing that around and doing a physically based um simulation doing a physically based um simulation doing a physically based um simulation of what's happening in the scene. If of what's happening in the scene. If of what's happening in the scene. If you're using a gaming engine that's you're using a gaming engine that's you're using a gaming engine that's obviously geared towards obviously geared towards obviously geared towards looking plausible to our human brains looking plausible to our human brains looking plausible to our human brains and taking as many shortcuts as it and taking as many shortcuts as it and taking as many shortcuts as it possibly can do, but pumping out those possibly can do, but pumping out those possibly can do, but pumping out those frames, 120 frames a second, rather than frames, 120 frames a second, rather than frames, 120 frames a second, rather than taking taking taking enough time to get the best image. So enough time to get the best image. So enough time to get the best image. So talk to me more about what the problem talk to me more about what the problem talk to me more about what the problem is that Sky Engine AI is solving and why is that Sky Engine AI is solving and why is that Sky Engine AI is solving and why synthetic data is important versus just synthetic data is important versus just synthetic data is important versus just consuming as much uh existing data.

consuming as much uh existing data. consuming as much uh existing data. Well, a lot of the problem with getting Well, a lot of the problem with getting Well, a lot of the problem with getting the data that you need is that it could the data that you need is that it could the data that you need is that it could be dangerous to get it. It could be be dangerous to get it. It could be be dangerous to get it. It could be illegal to get it. If you think about illegal to get it. If you think about illegal to get it. If you think about autonomous vehicles, you can't go around autonomous vehicles, you can't go around autonomous vehicles, you can't go around training and testing them by throwing training and testing them by throwing training and testing them by throwing people out in front of cars. Sometimes people out in front of cars. Sometimes people out in front of cars. Sometimes the thing that you want to detect is not the thing that you want to detect is not the thing that you want to detect is not going to occur very frequently. going to occur very frequently. going to occur very frequently. And if you're talking about defects in And if you're talking about defects in And if you're talking about defects in high value pieces of metal, for example, high value pieces of metal, for example, high value pieces of metal, for example, you don't want to have pumped out uh you don't want to have pumped out uh you don't want to have pumped out uh hundreds of those um in order to have hundreds of those um in order to have hundreds of those um in order to have collected the right amount of defect collected the right amount of defect collected the right amount of defect samples in order to be able to train samples in order to be able to train samples in order to be able to train your computer vision system. your computer vision system. your computer vision system. um because by then you've probably um because by then you've probably um because by then you've probably racked up quite a cost in order to racked up quite a cost in order to racked up quite a cost in order to generate that if you understand what generate that if you understand what generate that if you understand what they could look like and you can have they could look like and you can have they could look like and you can have them appear anywhere on that object, any them appear anywhere on that object, any them appear anywhere on that object, any size, any orientation. And you can do size, any orientation. And you can do size, any orientation. And you can do that with synthetic data where you that with synthetic data where you that with synthetic data where you basically built everything with computer basically built everything with computer basically built everything with computer graphics.

graphics. graphics. And you can also change the location of And you can also change the location of And you can also change the location of the camera very easily if there are the camera very easily if there are the camera very easily if there are going to be any variations in the going to be any variations in the going to be any variations in the lighting. You can basically parameterize lighting. You can basically parameterize lighting. You can basically parameterize that 3D scene with all of those that 3D scene with all of those that 3D scene with all of those variables that I just mentioned. And you variables that I just mentioned. And you variables that I just mentioned. And you can create a vast training database to can create a vast training database to can create a vast training database to make sure that the computer vision model make sure that the computer vision model make sure that the computer vision model will recognize any of those potential will recognize any of those potential will recognize any of those potential defects. And you can do that early and defects. And you can do that early and defects. And you can do that early and you can avoid having to wait until you can avoid having to wait until you can avoid having to wait until you've pumped out these um defective you've pumped out these um defective you've pumped out these um defective pieces of metal um which are costly. And pieces of metal um which are costly. And pieces of metal um which are costly. And uh yeah, it's uh saves you time, saves uh yeah, it's uh saves you time, saves uh yeah, it's uh saves you time, saves you money of course because you're not you money of course because you're not you money of course because you're not um generating these undetected def um generating these undetected def um generating these undetected def defects which might then go on further defects which might then go on further defects which might then go on further into the manufacturing process where into the manufacturing process where into the manufacturing process where you're spending more money and more you're spending more money and more you're spending more money and more money and more money until somebody money and more money until somebody money and more money until somebody realizes there's a defect and then all realizes there's a defect and then all realizes there's a defect and then all that work needs to be thrown away. So that work needs to be thrown away. So that work needs to be thrown away. So yeah, th those are the the situations.

yeah, th those are the the situations. yeah, th those are the the situations. Um, there's obviously Um, there's obviously Um, there's obviously hard to avoid talking about this, but hard to avoid talking about this, but hard to avoid talking about this, but there's obviously things like situations there's obviously things like situations there's obviously things like situations where you have a drone flying around, where you have a drone flying around, where you have a drone flying around, possibly needing to detect um the possibly needing to detect um the possibly needing to detect um the opposition's vehicles and things like opposition's vehicles and things like opposition's vehicles and things like that, you can't really gather that that, you can't really gather that that, you can't really gather that information very well. If you do fly information very well. If you do fly information very well. If you do fly your drone anywhere near the the enemy, your drone anywhere near the the enemy, your drone anywhere near the the enemy, they're going to shoot your drone down they're going to shoot your drone down they're going to shoot your drone down if they can do. Um, and you'll also only if they can do. Um, and you'll also only if they can do. Um, and you'll also only get one camera angle from that single get one camera angle from that single get one camera angle from that single drone. If you can recreate that in a 3D drone. If you can recreate that in a 3D drone. If you can recreate that in a 3D environment, you can create those images environment, you can create those images environment, you can create those images from any angle that you want by from any angle that you want by from any angle that you want by positioning that that drone's camera positioning that that drone's camera positioning that that drone's camera anywhere you want in the sky. So, it uh anywhere you want in the sky. So, it uh anywhere you want in the sky. So, it uh it's applicable all over. to be honest, it's applicable all over. to be honest, it's applicable all over. to be honest, the uh I'm looking at the use cases the uh I'm looking at the use cases the uh I'm looking at the use cases section of the Sky Engine AI website and section of the Sky Engine AI website and section of the Sky Engine AI website and there's a guy uh there's a a graphic of there's a guy uh there's a a graphic of there's a guy uh there's a a graphic of a guy driving a car and he's yawning and a guy driving a car and he's yawning and a guy driving a car and he's yawning and certainly someone falling asleep while certainly someone falling asleep while certainly someone falling asleep while driving a car would be something you'd driving a car would be something you'd driving a car would be something you'd want to be notified about. You'd want to want to be notified about. You'd want to want to be notified about. You'd want to have them shake the steering wheel or have them shake the steering wheel or have them shake the steering wheel or wake the guy up. So that makes me think wake the guy up. So that makes me think wake the guy up. So that makes me think that you're talking about like all of that you're talking about like all of that you're talking about like all of these edge cases where if you were to these edge cases where if you were to these edge cases where if you were to find real data of edge cases, you might find real data of edge cases, you might find real data of edge cases, you might watch thousands and thousands of hours watch thousands and thousands of hours watch thousands and thousands of hours of people driving happily to find the 20 of people driving happily to find the 20 of people driving happily to find the 20 times that someone falls asleep in their times that someone falls asleep in their times that someone falls asleep in their car and by then, of course, it's car and by then, of course, it's car and by then, of course, it's dangerous and those people are are not dangerous and those people are are not dangerous and those people are are not okay.

okay. okay. Yeah. And and of course, if you're doing Yeah. And and of course, if you're doing Yeah. And and of course, if you're doing that in the real world, you need to do that in the real world, you need to do that in the real world, you need to do that with a physical car on a physical that with a physical car on a physical that with a physical car on a physical road. Um, you could have somebody drive road. Um, you could have somebody drive road. Um, you could have somebody drive around a test circuit and and get them around a test circuit and and get them around a test circuit and and get them drunk. I'm not sure what the uh uh legal drunk. I'm not sure what the uh uh legal drunk. I'm not sure what the uh uh legal issues with that would be, but um I issues with that would be, but um I issues with that would be, but um I wouldn't be surprised if if somebody's wouldn't be surprised if if somebody's wouldn't be surprised if if somebody's tried to get footage that way, but it's tried to get footage that way, but it's tried to get footage that way, but it's not the same as having people on not the same as having people on not the same as having people on in in in the correct environment. in in in the correct environment. in in in the correct environment. And there was actually uh a Japanese And there was actually uh a Japanese And there was actually uh a Japanese company I came across a few years ago company I came across a few years ago company I came across a few years ago who put cameras in all of their uh cargo who put cameras in all of their uh cargo who put cameras in all of their uh cargo vans and did actually over something vans and did actually over something vans and did actually over something like 3 years capture their just a like 3 years capture their just a like 3 years capture their just a ridiculous amount of footage of their ridiculous amount of footage of their ridiculous amount of footage of their drivers when they got sleepy. So drivers when they got sleepy. So drivers when they got sleepy. So obviously 99% of what they captured was obviously 99% of what they captured was obviously 99% of what they captured was nobody got sleepy, everything was fine. nobody got sleepy, everything was fine. nobody got sleepy, everything was fine. and they they did actually capture that. and they they did actually capture that. and they they did actually capture that. But uh yeah, it's uh it's not a great But uh yeah, it's uh it's not a great But uh yeah, it's uh it's not a great way to do that. And obviously some of way to do that. And obviously some of way to do that. And obviously some of the drivers were were hurt during that.

the drivers were were hurt during that. the drivers were were hurt during that. And you know, that's the last thing you And you know, that's the last thing you And you know, that's the last thing you want. You want to be want. You want to be want. You want to be doing it in a preventative clever way so doing it in a preventative clever way so doing it in a preventative clever way so that you've uh got absolutely as much that you've uh got absolutely as much that you've uh got absolutely as much training data for your computer vision training data for your computer vision training data for your computer vision system as you need. system as you need. system as you need. Yeah. And then there's the uh the union Yeah. And then there's the uh the union Yeah. And then there's the uh the union kind of the vin diagram of all the kind of the vin diagram of all the kind of the vin diagram of all the different things that could be happening different things that could be happening different things that could be happening while that person's yawning. So for while that person's yawning. So for while that person's yawning. So for example, someone yawns and they happen example, someone yawns and they happen example, someone yawns and they happen to go underneath an under an overpass to go underneath an under an overpass to go underneath an under an overpass which causes a shadow on their face and which causes a shadow on their face and which causes a shadow on their face and there's a light from a passing car that there's a light from a passing car that there's a light from a passing car that flashes their brights at them. There flashes their brights at them. There flashes their brights at them. There could be a confluence of two or three could be a confluence of two or three could be a confluence of two or three edge cases that become a multiplier on edge cases that become a multiplier on edge cases that become a multiplier on top of each other. And that might never top of each other. And that might never top of each other. And that might never happen in the real world, but the one happen in the real world, but the one happen in the real world, but the one time it does happen is the one time you time it does happen is the one time you time it does happen is the one time you need your vision system to not fail. need your vision system to not fail. need your vision system to not fail. That can be synthesized. That can be synthesized. That can be synthesized. Yeah. Cuz that's a great point because Yeah. Cuz that's a great point because Yeah. Cuz that's a great point because if you or I were to look at the image, if you or I were to look at the image, if you or I were to look at the image, you would im your brain would you would im your brain would you would im your brain would immediately start throwing away all of immediately start throwing away all of immediately start throwing away all of those edge case um contributors that you those edge case um contributors that you those edge case um contributors that you just talked about. You know, a little just talked about. You know, a little just talked about. You know, a little bit of blue light from the side. Your bit of blue light from the side. Your bit of blue light from the side. Your brain would be like, "Yeah, not brain would be like, "Yeah, not brain would be like, "Yeah, not important." The important thing is important." The important thing is important." The important thing is there's a human face there. So you your there's a human face there. So you your there's a human face there. So you your brain would immediately start rejecting brain would immediately start rejecting brain would immediately start rejecting those uh bits and pieces. Whereas for a those uh bits and pieces. Whereas for a those uh bits and pieces. Whereas for a computer vision algorithm, you know, a computer vision algorithm, you know, a computer vision algorithm, you know, a patch of blue on somebody's face could patch of blue on somebody's face could patch of blue on somebody's face could make the difference between a um a false make the difference between a um a false make the difference between a um a false positive, false negative, a positive positive, false negative, a positive positive, false negative, a positive positive, a true positive. That's what I positive, a true positive. That's what I positive, a true positive. That's what I meant. Yeah.

meant. Yeah. meant. Yeah. Yeah. You know, I had a a guest on a Yeah. You know, I had a a guest on a Yeah. You know, I had a a guest on a couple of hundred episodes ago named couple of hundred episodes ago named couple of hundred episodes ago named Pelanomi Moa who does uh custom voice Pelanomi Moa who does uh custom voice Pelanomi Moa who does uh custom voice models in South Africa. And one of the models in South Africa. And one of the models in South Africa. And one of the things that she called out is that things that she called out is that things that she called out is that synthetic data can also be more ethical synthetic data can also be more ethical synthetic data can also be more ethical than regular data. As you call out, you than regular data. As you call out, you than regular data. As you call out, you there's privacy issues. She had voice there's privacy issues. She had voice there's privacy issues. She had voice models where they would actually hire models where they would actually hire models where they would actually hire actors and put on a pretend call center actors and put on a pretend call center actors and put on a pretend call center and they were all paid and they went and and they were all paid and they went and and they were all paid and they went and they generated hundreds and hundreds of they generated hundreds and hundreds of they generated hundreds and hundreds of hours, thousands of hours of call center hours, thousands of hours of call center hours, thousands of hours of call center data by faking it in the sense of data by faking it in the sense of data by faking it in the sense of synthesizing it, hiring people to synthesizing it, hiring people to synthesizing it, hiring people to pretend they were in a call center. and pretend they were in a call center. and pretend they were in a call center. and everybody wins because then you don't everybody wins because then you don't everybody wins because then you don't have the privacy issues of pulling that have the privacy issues of pulling that have the privacy issues of pulling that data. So it sounds like there's ethical data. So it sounds like there's ethical data. So it sounds like there's ethical reasons to use synthetic data as well. reasons to use synthetic data as well. reasons to use synthetic data as well. Yeah, I think I think also if you think Yeah, I think I think also if you think Yeah, I think I think also if you think about the the use case where you have about the the use case where you have about the the use case where you have public spaces public spaces public spaces um or even more critically public spaces um or even more critically public spaces um or even more critically public spaces during an incident where something uh during an incident where something uh during an incident where something uh dangerous or violent has happened and dangerous or violent has happened and dangerous or violent has happened and you therefore have the the rights to set you therefore have the the rights to set you therefore have the the rights to set up cameras to monitor the area then up cameras to monitor the area then up cameras to monitor the area then yeah, how do you train that? Do you get yeah, how do you train that? Do you get yeah, how do you train that? Do you get actors to run around in front of the actors to run around in front of the actors to run around in front of the camera or do you just set up cameras camera or do you just set up cameras camera or do you just set up cameras and, you know, surveil the public? You and, you know, surveil the public? You and, you know, surveil the public? You know, that that's not a great solution.

know, that that's not a great solution. know, that that's not a great solution. And I think uh putting synthetic humans And I think uh putting synthetic humans And I think uh putting synthetic humans in front of the camera, you can put the in front of the camera, you can put the in front of the camera, you can put the camera anywhere you want. You can change camera anywhere you want. You can change camera anywhere you want. You can change the lighting variables, you can change the lighting variables, you can change the lighting variables, you can change the clothes, you know, in a fraction of the clothes, you know, in a fraction of the clothes, you know, in a fraction of a second, run it again, get your data. a second, run it again, get your data. a second, run it again, get your data. Um, you've got ultimate flexibility Um, you've got ultimate flexibility Um, you've got ultimate flexibility there if you basically have a a 3D there if you basically have a a 3D there if you basically have a a 3D version of the environment. version of the environment. version of the environment. Now, when you're rendering for the real Now, when you're rendering for the real Now, when you're rendering for the real world, you want to render for the human world, you want to render for the human world, you want to render for the human eye and all of the things that we talked eye and all of the things that we talked eye and all of the things that we talked about that the human eye perceives as about that the human eye perceives as about that the human eye perceives as well as the uh the psychology of well as the uh the psychology of well as the uh the psychology of perception and how we perceive. Do you perception and how we perceive. Do you perception and how we perceive. Do you change rendering when the when the change rendering when the when the change rendering when the when the client is not someone with an eyeball, client is not someone with an eyeball, client is not someone with an eyeball, the client is a computer vision model? the client is a computer vision model? the client is a computer vision model? like does it look different? like does it look different? like does it look different? Uh ideally it should look as close as Uh ideally it should look as close as Uh ideally it should look as close as possible to the real example images that possible to the real example images that possible to the real example images that they've already collected. it uh should they've already collected. it uh should they've already collected. it uh should absolutely be as as realistic as absolutely be as as realistic as absolutely be as as realistic as possible because at the end of the day, possible because at the end of the day, possible because at the end of the day, if you're training your computer vision if you're training your computer vision if you're training your computer vision model on something that doesn't look model on something that doesn't look model on something that doesn't look quite right, it's got the uncanny valley quite right, it's got the uncanny valley quite right, it's got the uncanny valley thing going on or it looks a bit thing going on or it looks a bit thing going on or it looks a bit plastic, then you're training your model plastic, then you're training your model plastic, then you're training your model and basically creating a a domain gap, a and basically creating a a domain gap, a and basically creating a a domain gap, a reality gap there. And you should reality gap there. And you should reality gap there. And you should absolutely, you know, do your best to absolutely, you know, do your best to absolutely, you know, do your best to make things as close as possible to what make things as close as possible to what make things as close as possible to what the um computer vision model would the um computer vision model would the um computer vision model would ultimately have to deal with.

ultimately have to deal with. ultimately have to deal with. How do you then understand success? You How do you then understand success? You How do you then understand success? You render if we use the example of the render if we use the example of the render if we use the example of the gentleman yawning in his car and we gentleman yawning in his car and we gentleman yawning in his car and we create, you know, hundreds of hours of create, you know, hundreds of hours of create, you know, hundreds of hours of that and then we can now train a model that and then we can now train a model that and then we can now train a model on it. Do we then test it by using real on it. Do we then test it by using real on it. Do we then test it by using real examples to see if the synthetic data examples to see if the synthetic data examples to see if the synthetic data represented the uh the incident represented the uh the incident represented the uh the incident appropriately? appropriately? appropriately? Yeah, I think that is the the ideal Yeah, I think that is the the ideal Yeah, I think that is the the ideal situation to some extent. situation to some extent. situation to some extent. If you've got the computer vision model If you've got the computer vision model If you've got the computer vision model and it has a certain performance, it's and it has a certain performance, it's and it has a certain performance, it's it shouldn't well you could you could it shouldn't well you could you could it shouldn't well you could you could argue that it doesn't really matter how argue that it doesn't really matter how argue that it doesn't really matter how it got there. So if we got there with it got there. So if we got there with it got there. So if we got there with synthetic data rather than spending synthetic data rather than spending synthetic data rather than spending millions of pounds trying to collect millions of pounds trying to collect millions of pounds trying to collect real data, the performance is what real data, the performance is what real data, the performance is what matters. So if you have a large amount matters. So if you have a large amount matters. So if you have a large amount of synthetic data, you can generate of synthetic data, you can generate of synthetic data, you can generate that. You've only got a small amount of that. You've only got a small amount of that. You've only got a small amount of real data. You absolutely should be real data. You absolutely should be real data. You absolutely should be saving that for testing. Although you saving that for testing. Although you saving that for testing. Although you might need to add to that some of your might need to add to that some of your might need to add to that some of your synthetic data just to round out your synthetic data just to round out your synthetic data just to round out your tests so that you're covering all of the tests so that you're covering all of the tests so that you're covering all of the um all of the scenarios. And uh you know um all of the scenarios. And uh you know um all of the scenarios. And uh you know you would obviously be able to see you would obviously be able to see you would obviously be able to see whether there was a big difference whether there was a big difference whether there was a big difference between when you taste test on the real between when you taste test on the real between when you taste test on the real data that you've got and when you test data that you've got and when you test data that you've got and when you test on the synthetic examples that you've on the synthetic examples that you've on the synthetic examples that you've got. And there shouldn't be much of a got. And there shouldn't be much of a got. And there shouldn't be much of a difference.

difference. difference. Yeah, absolutely. Save all of your real Yeah, absolutely. Save all of your real Yeah, absolutely. Save all of your real data for the validation stage where data for the validation stage where data for the validation stage where you're testing your computer vision you're testing your computer vision you're testing your computer vision model. model. model. Is there a possibility of uh and I don't Is there a possibility of uh and I don't Is there a possibility of uh and I don't know if I'm using the right words here know if I'm using the right words here know if I'm using the right words here of of of poisoning the the data if you of of of poisoning the the data if you of of of poisoning the the data if you do the synthetic data wrong, could you do the synthetic data wrong, could you do the synthetic data wrong, could you potentially introduce some kind of bias potentially introduce some kind of bias potentially introduce some kind of bias that you didn't realize? Whether it be a that you didn't realize? Whether it be a that you didn't realize? Whether it be a bias uh a cultural bias or or more bias uh a cultural bias or or more bias uh a cultural bias or or more reasonably like a lighting bias like oh reasonably like a lighting bias like oh reasonably like a lighting bias like oh we assumed lighting was like this and it we assumed lighting was like this and it we assumed lighting was like this and it turns out that it's not and all of our turns out that it's not and all of our turns out that it's not and all of our synthetic data is not valid because of synthetic data is not valid because of synthetic data is not valid because of that biased assumption that we made. that biased assumption that we made. that biased assumption that we made. Well, it's of course it's absolutely Well, it's of course it's absolutely Well, it's of course it's absolutely possible. But because you are designing possible. But because you are designing possible. But because you are designing the environment and the parameterization the environment and the parameterization the environment and the parameterization of that, if you talk about for example of that, if you talk about for example of that, if you talk about for example ethnicity and you've say separated that ethnicity and you've say separated that ethnicity and you've say separated that into five core ethnicities, into five core ethnicities, into five core ethnicities, you can absolutely say 20% for each of you can absolutely say 20% for each of you can absolutely say 20% for each of these ethnicities because you want your these ethnicities because you want your these ethnicities because you want your computer vision model to have the same computer vision model to have the same computer vision model to have the same performance for each of these performance for each of these performance for each of these ethnicities rather than you going out ethnicities rather than you going out ethnicities rather than you going out there and saying, "Okay, well, I found, there and saying, "Okay, well, I found, there and saying, "Okay, well, I found, you know, So 50 50% of the people I I you know, So 50 50% of the people I I you know, So 50 50% of the people I I got to sign up to create our data were got to sign up to create our data were got to sign up to create our data were from this ethnicity. Another 30% were from this ethnicity. Another 30% were from this ethnicity. Another 30% were from this ethnicity and we only got 5% from this ethnicity and we only got 5% from this ethnicity and we only got 5% from this ethnicity. Then obviously your from this ethnicity. Then obviously your from this ethnicity. Then obviously your model is not going to be as robustly model is not going to be as robustly model is not going to be as robustly trained for the ethnicity where you only trained for the ethnicity where you only trained for the ethnicity where you only find found 5%. But because you're find found 5%. But because you're find found 5%. But because you're specifying this in the platform, you specifying this in the platform, you specifying this in the platform, you you're you're deciding what you need.

you're you're deciding what you need. you're you're deciding what you need. And if you find And if you find And if you find So you can remove bias through this So you can remove bias through this So you can remove bias through this technique. Well, if you find for some technique. Well, if you find for some technique. Well, if you find for some reason that your model is not performing reason that your model is not performing reason that your model is not performing so well with people of site with with so well with people of site with with so well with people of site with with people who have site correcting classes, people who have site correcting classes, people who have site correcting classes, you can go back and say okay well I need you can go back and say okay well I need you can go back and say okay well I need additional data for that and you can additional data for that and you can additional data for that and you can generate that additional data do some generate that additional data do some generate that additional data do some retraining and when you do your retraining and when you do your retraining and when you do your validation stage you should find that validation stage you should find that validation stage you should find that because you've used additional data the because you've used additional data the because you've used additional data the performance has gone up. So you can tune performance has gone up. So you can tune performance has gone up. So you can tune your performance and that's the your performance and that's the your performance and that's the flexibility of using synthetic data. flexibility of using synthetic data. flexibility of using synthetic data. Very cool. Now um there's among I'm I'm Very cool. Now um there's among I'm I'm Very cool. Now um there's among I'm I'm a web person and amongst the web there's a web person and amongst the web there's a web person and amongst the web there's concern about what they call browser concern about what they call browser concern about what they call browser monoculture which is there used to be monoculture which is there used to be monoculture which is there used to be many different renderers and uh you know many different renderers and uh you know many different renderers and uh you know there was like Firefox and Chrome and there was like Firefox and Chrome and there was like Firefox and Chrome and and and and Windows had Internet and and and Windows had Internet and and and Windows had Internet Explorer and now there's mostly just Explorer and now there's mostly just Explorer and now there's mostly just Chromium. Is there any concern for uh a Chromium. Is there any concern for uh a Chromium. Is there any concern for uh a rendering engine monoculture amongst rendering engine monoculture amongst rendering engine monoculture amongst gamers? It's just like well there's gamers? It's just like well there's gamers? It's just like well there's Unity uh you know do you need custom Unity uh you know do you need custom Unity uh you know do you need custom rendering engines in a non-gaming field rendering engines in a non-gaming field rendering engines in a non-gaming field or do you just is there just uh the or do you just is there just uh the or do you just is there just uh the confluence of a single rendering engine confluence of a single rendering engine confluence of a single rendering engine and then there'll be just one to rule and then there'll be just one to rule and then there'll be just one to rule them all or is this an environment that them all or is this an environment that them all or is this an environment that requires really custom work? Well, they requires really custom work? Well, they requires really custom work? Well, they tend to they tend to be proprietary just tend to they tend to be proprietary just tend to they tend to be proprietary just like the one at Sky Engine AI. And the like the one at Sky Engine AI. And the like the one at Sky Engine AI. And the reason for that is you need access to reason for that is you need access to reason for that is you need access to all of the code. And if you're going to

all of the code. And if you're going to all of the code. And if you're going to go beyond visible light, not just red, go beyond visible light, not just red, go beyond visible light, not just red, green, and blue, and you want to do near green, and blue, and you want to do near green, and blue, and you want to do near infrared, for example, infrared, for example, infrared, for example, ah, ah, ah, or if you want to do things like or if you want to do things like or if you want to do things like millimeter wave radar, which is very millimeter wave radar, which is very millimeter wave radar, which is very useful for it for child presence useful for it for child presence useful for it for child presence detection in vehicles. you you need your detection in vehicles. you you need your detection in vehicles. you you need your own renderer as a company to get stuck own renderer as a company to get stuck own renderer as a company to get stuck in there and get your R&D guys to uh do in there and get your R&D guys to uh do in there and get your R&D guys to uh do what is required in the real world for what is required in the real world for what is required in the real world for all of the sensors that are being uh all of the sensors that are being uh all of the sensors that are being uh created by the center manufacturers. created by the center manufacturers. created by the center manufacturers. Have you what have you found in your Have you what have you found in your Have you what have you found in your space understanding ver visual effects space understanding ver visual effects space understanding ver visual effects and putting things on screen for for and putting things on screen for for and putting things on screen for for entertainment and how is that translated entertainment and how is that translated entertainment and how is that translated into putting things on screen for AI into putting things on screen for AI into putting things on screen for AI consumption? consumption? consumption? That's a tricky question. Um although we That's a tricky question. Um although we That's a tricky question. Um although we to some extent we've we had we did talk to some extent we've we had we did talk to some extent we've we had we did talk about this a little bit and I I feel about this a little bit and I I feel about this a little bit and I I feel that VFX is all about telling the story.

that VFX is all about telling the story. that VFX is all about telling the story. Films are all about telling the story. Films are all about telling the story. Films are all about telling the story. Whereas, Whereas, Whereas, as we were discussing earlier, it's it as we were discussing earlier, it's it as we were discussing earlier, it's it for training a computer vision model, it for training a computer vision model, it for training a computer vision model, it really is about the whole frame because really is about the whole frame because really is about the whole frame because something in that frame that you've something in that frame that you've something in that frame that you've ignored as a human and and have ignored as a human and and have ignored as a human and and have discounted as not important might well discounted as not important might well discounted as not important might well affect the training of your computer affect the training of your computer affect the training of your computer vision model. M vision model. M vision model. M um although this does remind me of um although this does remind me of um although this does remind me of working on a shot where it was a studio working on a shot where it was a studio working on a shot where it was a studio shot and in the foreground there was shot and in the foreground there was shot and in the foreground there was this rock and uh my supervisor said to this rock and uh my supervisor said to this rock and uh my supervisor said to me, "Yeah, that that rock's not rocky me, "Yeah, that that rock's not rocky me, "Yeah, that that rock's not rocky enough." And I was like, "Oh, sorry. enough." And I was like, "Oh, sorry. enough." And I was like, "Oh, sorry. What What do you mean that's a rock?" Uh What What do you mean that's a rock?" Uh What What do you mean that's a rock?" Uh and he was like, "Yeah, no, it it and he was like, "Yeah, no, it it and he was like, "Yeah, no, it it doesn't look rocky enough. Can you can doesn't look rocky enough. Can you can doesn't look rocky enough. Can you can you add some extra detail into because I you add some extra detail into because I you add some extra detail into because I had this flat patch and and having a had this flat patch and and having a had this flat patch and and having a look at it and having a think I was look at it and having a think I was look at it and having a think I was like, "Yeah, that it's not really like, "Yeah, that it's not really like, "Yeah, that it's not really chiming as a rocky rock." So, I painted chiming as a rocky rock." So, I painted chiming as a rocky rock." So, I painted up some bits of light and dark and uh up some bits of light and dark and uh up some bits of light and dark and uh you know, multiplied the colors through you know, multiplied the colors through you know, multiplied the colors through that and gave it a bit more texture and that and gave it a bit more texture and that and gave it a bit more texture and uh made it look more rocky and then uh made it look more rocky and then uh made it look more rocky and then everyone was happy.

everyone was happy. everyone was happy. That's that's so funny that someone That's that's so funny that someone That's that's so funny that someone would have a perception like that. that would have a perception like that. that would have a perception like that. that makes me want to go and look at your makes me want to go and look at your makes me want to go and look at your IMDb page and try to chase that chase IMDb page and try to chase that chase IMDb page and try to chase that chase that shot down and figure out when that shot down and figure out when that shot down and figure out when exactly that happened. But in that case exactly that happened. But in that case exactly that happened. But in that case there that the supervisor or the there that the supervisor or the there that the supervisor or the director above them had a had a director above them had a had a director above them had a had a perspective and the idea that you can perspective and the idea that you can perspective and the idea that you can make that perspective uh possible is uh make that perspective uh possible is uh make that perspective uh possible is uh is amazing. Those are the kind of shots is amazing. Those are the kind of shots is amazing. Those are the kind of shots that you don't even realize. There's so that you don't even realize. There's so that you don't even realize. There's so much really great work happening in CGI much really great work happening in CGI much really great work happening in CGI right now where they improve a shot or right now where they improve a shot or right now where they improve a shot or they juzj up a shot a little bit but you they juzj up a shot a little bit but you they juzj up a shot a little bit but you don't know why. it just it's just don't know why. it just it's just don't know why. it just it's just better. But we don't perceive them as better. But we don't perceive them as better. But we don't perceive them as visual effects shots, you know. visual effects shots, you know. visual effects shots, you know. Yeah. So, it's all about where your Yeah. So, it's all about where your Yeah. So, it's all about where your eyeballs go in in the shot. And, you eyeballs go in in the shot. And, you eyeballs go in in the shot. And, you know, the the rock was definitely not know, the the rock was definitely not know, the the rock was definitely not telling the story. So, it it needed it telling the story. So, it it needed it telling the story. So, it it needed it needed to be ignorable. needed to be ignorable. needed to be ignorable. Yeah, that is definitely a kind of Yeah, that is definitely a kind of Yeah, that is definitely a kind of artistry. Uh indeed. Um so, back to artistry. Uh indeed. Um so, back to artistry. Uh indeed. Um so, back to synthetic data, this is something that synthetic data, this is something that synthetic data, this is something that is uh way cheaper as well. like you can is uh way cheaper as well. like you can is uh way cheaper as well. like you can make ton of data at the same cost that make ton of data at the same cost that make ton of data at the same cost that it would have been to go and collect the it would have been to go and collect the it would have been to go and collect the data. Do you do all the work is data. Do you do all the work is data. Do you do all the work is synthetic or do you do when you feed a synthetic or do you do when you feed a synthetic or do you do when you feed a computer vision model are you is there a computer vision model are you is there a computer vision model are you is there a percentage of like well 80% is synthetic percentage of like well 80% is synthetic percentage of like well 80% is synthetic and 20% is real and how does one make and 20% is real and how does one make and 20% is real and how does one make decisions like that?

decisions like that? decisions like that? Well I think as we were discussing Well I think as we were discussing Well I think as we were discussing before if you've got real data you before if you've got real data you before if you've got real data you really should save it for your testing really should save it for your testing really should save it for your testing for your validation stage. Okay, that's for your validation stage. Okay, that's for your validation stage. Okay, that's validation validation validation because if your model has been trained because if your model has been trained because if your model has been trained and it's gotten certain performance, and it's gotten certain performance, and it's gotten certain performance, um you could mix in some real data if um you could mix in some real data if um you could mix in some real data if you've got it, but you should really you've got it, but you should really you've got it, but you should really just use the flexibility of good quality just use the flexibility of good quality just use the flexibility of good quality synthetic data for that. So, you're synthetic data for that. So, you're synthetic data for that. So, you're covering all all of your edge cases. covering all all of your edge cases. covering all all of your edge cases. Mhm. And then what is what is transfer Mhm. And then what is what is transfer Mhm. And then what is what is transfer learning? Well, that is something that learning? Well, that is something that learning? Well, that is something that you see a lot with uh LLM models you see a lot with uh LLM models you see a lot with uh LLM models and if you've already trained a model and if you've already trained a model and if you've already trained a model and it's quite good and you want to and it's quite good and you want to and it's quite good and you want to basically suck out some of the brains of basically suck out some of the brains of basically suck out some of the brains of that into your model which might have a that into your model which might have a that into your model which might have a different architecture different architecture different architecture then that's something that's something then that's something that's something then that's something that's something you can do because you can do because you can do because I think the and I'm not and I'm I think the and I'm not and I'm I think the and I'm not and I'm definitely not not an expert on this definitely not not an expert on this definitely not not an expert on this because I don't create um AI models because I don't create um AI models because I don't create um AI models myself, but if you've got a model where myself, but if you've got a model where myself, but if you've got a model where you can essentially pose questions and you can essentially pose questions and you can essentially pose questions and get answers, then that's that's get answers, then that's that's get answers, then that's that's essentially training data.

essentially training data. essentially training data. Okay. So, if you have a problem and you Okay. So, if you have a problem and you Okay. So, if you have a problem and you want to solve it with computer vision, want to solve it with computer vision, want to solve it with computer vision, you can use a base model and then you can use a base model and then you can use a base model and then transfer transfer transfer building a new model on top of the building a new model on top of the building a new model on top of the previous model. like you're you're b previous model. like you're you're b previous model. like you're you're b building upon the work of others and building upon the work of others and building upon the work of others and then adding a delta of the things that then adding a delta of the things that then adding a delta of the things that matter and then perhaps adding synthetic matter and then perhaps adding synthetic matter and then perhaps adding synthetic data on top of that to get the result data on top of that to get the result data on top of that to get the result that you want. that you want. that you want. Yeah, certainly possible. Although I Yeah, certainly possible. Although I Yeah, certainly possible. Although I think a lot of the time what you want to think a lot of the time what you want to think a lot of the time what you want to point your camera or sensor at can be point your camera or sensor at can be point your camera or sensor at can be something very specific which you won't something very specific which you won't something very specific which you won't find in a model which has been being find in a model which has been being find in a model which has been being trained on things in general you know trained on things in general you know trained on things in general you know like the segment anything model. I think like the segment anything model. I think like the segment anything model. I think if you are doing this for LLMs, however, if you are doing this for LLMs, however, if you are doing this for LLMs, however, then it the the common factor there is then it the the common factor there is then it the the common factor there is words and human communication and you're words and human communication and you're words and human communication and you're less likely to be in a novel situation less likely to be in a novel situation less likely to be in a novel situation unless you're going for something hyper unless you're going for something hyper unless you're going for something hyper specific like I don't know 18th century specific like I don't know 18th century specific like I don't know 18th century um legal opinions or something like um legal opinions or something like um legal opinions or something like that. There's there's so much that. There's there's so much that. There's there's so much interesting stuff though as we get interesting stuff though as we get interesting stuff though as we get towards the end of the podcast that can towards the end of the podcast that can towards the end of the podcast that can be used with with vision, computer be used with with vision, computer be used with with vision, computer vision, particularly like in spaces like vision, particularly like in spaces like vision, particularly like in spaces like healthcare around personal safety, human healthcare around personal safety, human healthcare around personal safety, human safety. You know, uh I've seen healthc safety. You know, uh I've seen healthc safety. You know, uh I've seen healthc care models that can detect and predict care models that can detect and predict care models that can detect and predict if someone's going to fall out of their if someone's going to fall out of their if someone's going to fall out of their bed. So, you know, old folks, they're in bed. So, you know, old folks, they're in bed. So, you know, old folks, they're in bed, they're in hospital, they start to bed, they're in hospital, they start to bed, they're in hospital, they start to move a certain way, and they might fall move a certain way, and they might fall move a certain way, and they might fall out of the bed and break a hip. and

out of the bed and break a hip. and out of the bed and break a hip. and preventing that or alerting the nurse preventing that or alerting the nurse preventing that or alerting the nurse with with a a reasonable amount of uh with with a a reasonable amount of uh with with a a reasonable amount of uh accuracy 5 minutes before that happens accuracy 5 minutes before that happens accuracy 5 minutes before that happens using computer vision especially using computer vision especially using computer vision especially computer vision over a temporal time computer vision over a temporal time computer vision over a temporal time like not just I observed it in this like not just I observed it in this like not just I observed it in this frame but I observed it over a period of frame but I observed it over a period of frame but I observed it over a period of time is hugely valuable time is hugely valuable time is hugely valuable now that that's that's uh reminds me of now that that's that's uh reminds me of now that that's that's uh reminds me of um I think there is an app out there for um I think there is an app out there for um I think there is an app out there for blind people they can point their phone blind people they can point their phone blind people they can point their phone camera camera camera Mhm. um and get a description of what's Mhm. um and get a description of what's Mhm. um and get a description of what's in front of of of the phone. And I think in front of of of the phone. And I think in front of of of the phone. And I think that's really amazing. that's really amazing. that's really amazing. That's so funny because uh I had the That's so funny because uh I had the That's so funny because uh I had the gentleman uh on Sakip Shik who developed gentleman uh on Sakip Shik who developed gentleman uh on Sakip Shik who developed the first version of that on the podcast the first version of that on the podcast the first version of that on the podcast uh 100 episodes ago and now that's uh 100 episodes ago and now that's uh 100 episodes ago and now that's becoming commoditized like if you became becoming commoditized like if you became becoming commoditized like if you became blind today that you would expect that blind today that you would expect that blind today that you would expect that level of of uh functionality. And I level of of uh functionality. And I level of of uh functionality. And I think that's so interesting that what think that's so interesting that what think that's so interesting that what was absolutely revolutionary 10 years was absolutely revolutionary 10 years was absolutely revolutionary 10 years ago would now be table stakes and I ago would now be table stakes and I ago would now be table stakes and I would hopefully have a a set of meta would hopefully have a a set of meta would hopefully have a a set of meta glasses with a little camera and it'd be glasses with a little camera and it'd be glasses with a little camera and it'd be whispering in my ear and letting me know whispering in my ear and letting me know whispering in my ear and letting me know what's going on in front of me.

what's going on in front of me. what's going on in front of me. Well, I I absolutely think that's that's Well, I I absolutely think that's that's Well, I I absolutely think that's that's where everything's going. Um intelligent where everything's going. Um intelligent where everything's going. Um intelligent smart glasses and you don't have to poke smart glasses and you don't have to poke smart glasses and you don't have to poke at a rectangle of electronics um to get at a rectangle of electronics um to get at a rectangle of electronics um to get it to do things. it then becomes it to do things. it then becomes it to do things. it then becomes something more natural. And also, if something more natural. And also, if something more natural. And also, if you're wearing glasses, then the AI you're wearing glasses, then the AI you're wearing glasses, then the AI model knows what you're looking at and model knows what you're looking at and model knows what you're looking at and maybe even do, you know, tracking of maybe even do, you know, tracking of maybe even do, you know, tracking of your eyeballs and knows knows what's in your eyeballs and knows knows what's in your eyeballs and knows knows what's in front of you, but also knows what you're front of you, but also knows what you're front of you, but also knows what you're paying attention to. If it can read your paying attention to. If it can read your paying attention to. If it can read your mind, then uh yeah, mind, then uh yeah, mind, then uh yeah, I think that's just a bridge too far, I think that's just a bridge too far, I think that's just a bridge too far, but I would like smart glasses at some but I would like smart glasses at some but I would like smart glasses at some point. point. point. So, so folks can learn more about the So, so folks can learn more about the So, so folks can learn more about the cool work that's happening at SkyEngine cool work that's happening at SkyEngine cool work that's happening at SkyEngine at skyengine.ai. at skyengine.ai. at skyengine.ai. You can go and learn more. And uh thank You can go and learn more. And uh thank You can go and learn more. And uh thank you so much Dr. Malcudter for spending you so much Dr. Malcudter for spending you so much Dr. Malcudter for spending time with me today. time with me today. time with me today. Well, that was a genuine pleasure, Well, that was a genuine pleasure, Well, that was a genuine pleasure, Scott. Scott. Scott. This has been another episode of Hansel This has been another episode of Hansel This has been another episode of Hansel Minutes and we'll see you again next Minutes and we'll see you again next Minutes and we'll see you again next week.